2024-04-26T08:22:07.684 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T08:22:07.684 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} --name teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639 --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639/orig.config.yaml 2024-04-26T08:22:07.729 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T08:22:07.817 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639 branch: reef description: fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672639' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: 'install ceph/pacific latest ' name: teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: bluestore warn on legacy statfs: false bluestore warn on no per pool omap: false mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 mon warn on osd down out interval zero: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - scrub mismatch - ScrubResult - wrongly marked - \(POOL_APP_NOT_ENABLED\) - \(SLOW_OPS\) - overall HEALTH_ - \(MON_MSGR2_NOT_ENABLED\) - slow request sha1: 19b26365dac54ee709df8c5dc2a7b7e89a50aba5 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 19b26365dac54ee709df8c5dc2a7b7e89a50aba5 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: dc093b5e41f728edab6e0330295fdf9156d31664 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 seed: 7931 sha1: 19b26365dac54ee709df8c5dc2a7b7e89a50aba5 sleep_before_teardown: 0 subset: 126/512 suite: fs suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_dc093b5e41f728edab6e0330295fdf9156d31664/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: dc093b5e41f728edab6e0330295fdf9156d31664 targets: smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAWlc2kZ/DO1NfCuDq1WvvsV03cE++vlE+/MHCAreg73mQ6TuDVK3Ctt9MmSJJpn8zoPeSeXJ6LxMHohlkrJac8= tasks: - install: branch: pacific exclude_packages: - librados3 - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - cephadm - ceph-volume extra_packages: - librados2 - print: '**** done installing pacific' - ceph: conf: global: mon warn on pool no app: false ms bind msgr2: false log-ignorelist: - overall HEALTH_ - \(FS_ - \(MDS_ - \(OSD_ - \(MON_DOWN\) - \(CACHE_POOL_ - \(POOL_ - \(MGR_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - Monitor daemon marked osd - Behind on trimming - Manager daemon - exec: osd.0: - ceph osd set-require-min-compat-client pacific - print: '**** done ceph' - print: '*** upgrading, no cephfs present' - exec: mon.a: - ceph fs dump - install.upgrade: mon.a: null - print: '**** done install.upgrade' - ceph.restart: daemons: - mon.* - mgr.* mon-health-to-clog: false wait-for-healthy: false - ceph.healthy: null - ceph.restart: daemons: - osd.* wait-for-healthy: false wait-for-osds-up: true - exec: mon.a: - ceph versions - ceph osd dump -f json-pretty - ceph fs dump - ceph osd require-osd-release quincy - for f in `ceph osd pool ls` ; do ceph osd pool set $f pg_autoscale_mode off ; done - ceph.healthy: null - print: '**** done ceph.restart' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_22:24:15 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T08:22:07.817 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_dc093b5e41f728edab6e0330295fdf9156d31664/qa; will attempt to use it 2024-04-26T08:22:07.818 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_dc093b5e41f728edab6e0330295fdf9156d31664/qa/tasks 2024-04-26T08:22:07.818 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T08:22:07.820 INFO:teuthology.task.internal:Checking packages... 2024-04-26T08:22:07.850 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '19b26365dac54ee709df8c5dc2a7b7e89a50aba5' 2024-04-26T08:22:07.850 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T08:22:07.850 INFO:teuthology.packaging:ref: None 2024-04-26T08:22:07.850 INFO:teuthology.packaging:tag: None 2024-04-26T08:22:07.850 INFO:teuthology.packaging:branch: reef 2024-04-26T08:22:07.850 INFO:teuthology.packaging:sha1: 19b26365dac54ee709df8c5dc2a7b7e89a50aba5 2024-04-26T08:22:07.851 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef 2024-04-26T08:22:08.084 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1258.g7d2137f0 2024-04-26T08:22:08.085 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T08:22:08.092 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T08:22:08.092 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T08:22:08.099 INFO:teuthology.task.internal:Saving configuration 2024-04-26T08:22:08.115 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T08:22:08.121 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T08:22:08.155 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 08:16:59.501868', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAWlc2kZ/DO1NfCuDq1WvvsV03cE++vlE+/MHCAreg73mQ6TuDVK3Ctt9MmSJJpn8zoPeSeXJ6LxMHohlkrJac8='} 2024-04-26T08:22:08.155 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T08:22:08.161 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-04-26T08:22:08.162 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T08:22:08.215 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcad3ce8160>, signals=[15]) 2024-04-26T08:22:08.215 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T08:22:08.221 INFO:teuthology.task.internal:Opening connections... 2024-04-26T08:22:08.221 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T08:22:08.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T08:22:08.313 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T08:22:08.319 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-04-26T08:22:08.383 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-04-26T08:22:08.384 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-04-26T08:22:08.445 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-04-26T08:22:08.445 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="8" 2024-04-26T08:22:08.445 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-04-26T08:22:08.445 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-04-26T08:22:08.445 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="8" 2024-04-26T08:22:08.445 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el8" 2024-04-26T08:22:08.445 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-26T08:22:08.445 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-04-26T08:22:08.446 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-26T08:22:08.446 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-04-26T08:22:08.446 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-26T08:22:08.446 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-26T08:22:08.446 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-26T08:22:08.446 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-04-26T08:22:08.481 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T08:22:08.490 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T08:22:08.590 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T08:22:08.590 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-04-26T08:22:08.610 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T08:22:08.622 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T08:22:08.622 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-04-26T08:22:08.679 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T08:22:08.723 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T08:22:08.738 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-26T08:22:08.738 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-26T08:22:08.738 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T08:22:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T08:22:08.759 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-04-26T08:22:08.819 INFO:teuthology.orchestra.run.smithi194.stdout:4.18.0-489.el8.x86_64 2024-04-26T08:22:08.819 INFO:teuthology.task.kernel:Running kernel on smithi194: 4.18.0-489.el8.x86_64 2024-04-26T08:22:08.820 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-04-26T08:22:14.553 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - AppStream 8.9 MB/s | 28 MB 00:03 2024-04-26T08:28:17.910 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - BaseOS 30 kB/s | 10 MB 05:58 2024-04-26T08:28:19.885 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras 50 kB/s | 18 kB 00:00 2024-04-26T08:28:20.452 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 7.7 kB 00:00 2024-04-26T08:28:20.720 INFO:teuthology.orchestra.run.smithi194.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-04-26T08:28:22.312 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-04-26T08:28:26.104 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 580 kB/s | 24 kB 00:00 2024-04-26T08:28:29.126 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-26T08:28:29.126 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-26T08:28:29.242 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T08:28:29.242 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout:Install 3 Packages 2024-04-26T08:28:29.243 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T08:28:29.244 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 90 M 2024-04-26T08:28:29.244 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 96 M 2024-04-26T08:28:29.244 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T08:39:04.340 INFO:teuthology.orchestra.run.smithi194.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 17 kB/s | 10 MB 10:34 2024-04-26T09:03:01.146 INFO:teuthology.orchestra.run.smithi194.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 18 kB/s | 36 MB 34:31 2024-04-26T09:12:44.912 INFO:teuthology.orchestra.run.smithi194.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 17 kB/s | 44 MB 44:15 2024-04-26T09:12:44.912 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T09:12:44.912 INFO:teuthology.orchestra.run.smithi194.stdout:Total 35 kB/s | 90 MB 44:15 2024-04-26T09:12:45.597 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T09:12:46.024 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T09:12:46.024 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T09:12:47.580 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T09:12:47.654 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T09:12:50.761 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T09:12:51.315 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-26T09:12:54.904 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-26T09:12:55.348 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-26T09:12:58.787 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-26T09:12:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-26T09:13:21.016 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-26T09:13:21.038 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-26T09:13:21.389 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-26T09:13:21.390 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-26T09:13:21.390 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-26T09:13:21.997 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-26T09:13:21.998 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:13:21.998 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T09:13:21.998 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-26T09:13:21.998 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-26T09:13:21.998 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:13:21.998 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:13:22.140 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-04-26T09:13:22.357 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-26T09:13:22.357 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-26T09:13:22.357 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-26T09:13:22.357 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T09:13:22.358 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T09:13:22.358 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-26T09:13:22.358 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T09:13:22.358 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-04-26T09:13:22.571 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-26T09:13:22.571 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-26T09:13:22.571 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-04-26T09:13:22.605 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-04-26T09:13:22.605 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-04-26T09:13:22.605 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.02 2024-04-26T09:13:22.605 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 129.el8 2024-04-26T09:13:22.605 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-04-26T09:13:22.605 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-26T09:13:22.605 INFO:teuthology.orchestra.run.smithi194.stdout:Group : System Environment/Base 2024-04-26T09:13:22.605 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 9265586 2024-04-26T09:13:22.605 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-04-26T09:13:22.605 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-26T09:13:22.605 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:Relocations : (not relocatable) 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : CentOS BuildSystem 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:13:22.606 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-04-26T09:13:22.607 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-26T09:13:22.607 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 4.18.0-552.el8.x86_64 2024-04-26T09:13:22.608 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-26T09:13:23.892 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-04-26T09:13:26.185 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-04-26T09:13:26.187 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-04-26T09:13:26.222 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.BaX0dEOTUH 2024-04-26T09:13:26.222 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BaX0dEOTUH 2024-04-26T09:13:26.303 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.BaX0dEOTUH 2024-04-26T09:13:26.456 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.BaX0dEOTUH is 6KB 2024-04-26T09:13:26.505 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.BaX0dEOTUH 2024-04-26T09:13:26.523 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-04-26T09:13:26.594 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-26T09:13:26.595 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-26T09:13:26.595 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-26T09:13:26.595 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-26T09:13:26.596 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-26T09:13:26.758 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-04-26T09:13:56.782 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-26T09:13:56.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-26T09:13:56.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T09:14:15.199 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-04-26T09:14:24.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-26T09:14:24.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T09:14:27.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-04-26T09:14:39.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-26T09:14:39.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T09:14:57.694 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-04-26T09:15:12.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-26T09:15:12.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T09:15:13.704 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-04-26T09:15:14.989 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-26T09:15:14.990 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-26T09:15:14.993 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-04-26T09:15:15.010 INFO:teuthology.orchestra.run.smithi194.stdout:4.18.0-552.el8.x86_64 2024-04-26T09:15:15.010 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-26T09:15:15.010 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T09:15:15.010 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-26T09:15:16.014 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-26T09:15:16.014 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T09:15:16.095 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-04-26T09:15:16.111 DEBUG:teuthology.parallel:result is None 2024-04-26T09:15:16.112 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T09:15:16.120 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T09:15:16.120 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T09:15:16.139 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T09:15:16.146 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T09:15:16.194 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T09:15:16.194 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T09:15:16.226 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T09:15:16.233 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T09:15:16.233 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T09:15:16.322 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T09:15:16.332 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T09:15:16.333 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T09:15:16.341 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T09:15:16.341 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T09:15:16.367 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T09:15:16.376 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T09:15:16.377 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T09:15:16.425 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T09:15:16.507 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T09:15:16.563 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T09:15:16.644 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T09:15:16.690 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:15:16.690 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T09:15:16.760 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-04-26T09:15:16.856 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T09:15:17.200 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T09:15:17.208 INFO:teuthology.task.internal:Starting timer... 2024-04-26T09:15:17.208 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T09:15:17.218 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T09:15:17.227 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-26T09:15:17.228 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-04-26T09:15:17.305 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-04-26T09:15:17.307 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T09:15:17.307 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-04-26T09:15:17.343 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-04-26T09:15:17.343 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-04-26T09:15:17.343 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-26T09:15:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T09:15:17.410 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T09:15:17.411 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-04-26T09:15:17.479 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T09:15:17.488 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T09:15:17.505 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T09:15:17.506 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T09:26:30.534 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-04-26T09:26:30.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-26T09:26:30.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T09:26:30.612 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-04-26T09:26:30.681 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-26T09:26:30.681 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T09:26:30.697 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T09:26:30.697 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T09:26:30.698 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T09:26:30.752 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-26T09:26:30.766 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-26T09:26:30.832 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-04-26T09:26:30.844 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-04-26T09:26:30.858 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-26T09:26:30.871 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-26T09:26:30.932 INFO:teuthology.orchestra.run.smithi194.stderr:bash: ntpq: command not found 2024-04-26T09:26:30.935 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T09:26:30.935 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-04-26T09:26:30.935 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T09:26:30.935 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T09:26:30.935 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T09:26:30.936 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T09:26:30.936 INFO:teuthology.run_tasks:Running task install... 2024-04-26T09:26:30.945 DEBUG:teuthology.task.install:project ceph 2024-04-26T09:26:30.945 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '19b26365dac54ee709df8c5dc2a7b7e89a50aba5'}} 2024-04-26T09:26:30.945 DEBUG:teuthology.task.install:config {'branch': 'pacific', 'exclude_packages': ['librados3', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'cephadm', 'ceph-volume'], 'extra_packages': ['librados2'], 'flavor': 'default', 'sha1': '19b26365dac54ee709df8c5dc2a7b7e89a50aba5'} 2024-04-26T09:26:30.945 INFO:teuthology.task.install:Using flavor: default 2024-04-26T09:26:30.951 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T09:26:30.951 INFO:teuthology.task.install:extra packages: ['librados2'] 2024-04-26T09:26:30.951 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['librados3', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'cephadm', 'ceph-volume'], 'extra_packages': ['librados2'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '19b26365dac54ee709df8c5dc2a7b7e89a50aba5', 'tag': None, 'wait_for_package': False} 2024-04-26T09:26:30.952 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T09:26:30.952 INFO:teuthology.packaging:ref: None 2024-04-26T09:26:30.952 INFO:teuthology.packaging:tag: None 2024-04-26T09:26:30.952 INFO:teuthology.packaging:branch: pacific 2024-04-26T09:26:30.952 INFO:teuthology.packaging:sha1: 19b26365dac54ee709df8c5dc2a7b7e89a50aba5 2024-04-26T09:26:30.952 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-04-26T09:26:31.167 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/ 2024-04-26T09:26:31.167 INFO:teuthology.task.install.rpm:Package version is 16.2.15-0 2024-04-26T09:26:31.203 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/618f440892089921c3e944a991122ddc44e60516/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T09:26:31.203 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:26:31.203 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T09:26:31.231 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, librados2 on remote rpm x86_64 2024-04-26T09:26:31.231 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T09:26:31.231 INFO:teuthology.packaging:ref: None 2024-04-26T09:26:31.231 INFO:teuthology.packaging:tag: None 2024-04-26T09:26:31.231 INFO:teuthology.packaging:branch: pacific 2024-04-26T09:26:31.231 INFO:teuthology.packaging:sha1: 19b26365dac54ee709df8c5dc2a7b7e89a50aba5 2024-04-26T09:26:31.232 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T09:26:31.299 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T09:26:31.377 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T09:26:31.443 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-04-26T09:26:31.444 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-04-26T09:26:31.710 INFO:teuthology.orchestra.run.smithi194.stdout:78 files removed 2024-04-26T09:26:31.738 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw 2024-04-26T09:26:32.214 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 1.1 MB/s | 76 kB 00:00 2024-04-26T09:26:32.306 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 284 kB/s | 15 kB 00:00 2024-04-26T09:26:32.407 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-04-26T09:32:39.555 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - AppStream 78 kB/s | 28 MB 06:07 2024-04-26T09:32:44.978 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - BaseOS 21 MB/s | 10 MB 00:00 2024-04-26T09:32:46.962 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras 44 kB/s | 18 kB 00:00 2024-04-26T09:32:47.296 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras common packages 34 kB/s | 7.7 kB 00:00 2024-04-26T09:32:48.497 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - PowerTools 4.5 MB/s | 5.0 MB 00:01 2024-04-26T09:32:49.907 INFO:teuthology.orchestra.run.smithi194.stdout:Copr repo for python3-asyncssh owned by ceph 7.1 kB/s | 3.5 kB 00:00 2024-04-26T09:32:51.016 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2024-04-26T09:32:54.875 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 218 kB/s | 24 kB 00:00 2024-04-26T09:32:58.546 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:32:58.549 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:32:58.549 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-04-26T09:32:58.549 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:32:58.549 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T09:32:58.549 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:16.2.15-0.el8 ceph 12 M 2024-04-26T09:32:58.549 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-04-26T09:32:58.549 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:16.2.15-0.el8 ceph 3.7 M 2024-04-26T09:32:58.549 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:16.2.15-0.el8 ceph 4.2 M 2024-04-26T09:32:58.549 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:16.2.15-0.el8 ceph 6.1 M 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:16.2.15-0.el8 ceph 24 M 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:16.2.15-0.el8 ceph 26 k 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:16.2.15-0.el8 ceph 825 k 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:16.2.15-0.el8 ceph 494 k 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:16.2.15-0.el8 ceph 3.8 M 2024-04-26T09:32:58.550 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-04-26T09:32:58.551 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-04-26T09:32:58.551 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:16.2.15-0.el8 ceph 45 k 2024-04-26T09:32:58.551 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:16.2.15-0.el8 ceph 99 k 2024-04-26T09:32:58.551 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:16.2.15-0.el8 ceph 215 k 2024-04-26T09:32:58.551 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-04-26T09:32:58.551 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:16.2.15-0.el8 ceph 389 k 2024-04-26T09:32:58.551 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:16.2.15-0.el8 ceph 369 k 2024-04-26T09:32:58.551 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:16.2.15-0.el8 ceph 114 k 2024-04-26T09:32:58.551 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:32:58.551 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T09:32:58.551 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:32:58.552 INFO:teuthology.orchestra.run.smithi194.stdout:Install 21 Packages 2024-04-26T09:32:58.552 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 2 Packages 2024-04-26T09:32:58.552 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:32:58.552 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 58 M 2024-04-26T09:32:58.552 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T09:32:59.183 INFO:teuthology.orchestra.run.smithi194.stdout:(1/23): ceph-base-16.2.15-0.el8.x86_64.rpm 45 MB/s | 6.1 MB 00:00 2024-04-26T09:32:59.208 INFO:teuthology.orchestra.run.smithi194.stdout:(2/23): ceph-selinux-16.2.15-0.el8.x86_64.rpm 1.0 MB/s | 26 kB 00:00 2024-04-26T09:32:59.259 INFO:teuthology.orchestra.run.smithi194.stdout:(3/23): libcephfs2-16.2.15-0.el8.x86_64.rpm 16 MB/s | 825 kB 00:00 2024-04-26T09:32:59.292 INFO:teuthology.orchestra.run.smithi194.stdout:(4/23): libradosstriper1-16.2.15-0.el8.x86_64.r 14 MB/s | 494 kB 00:00 2024-04-26T09:32:59.376 INFO:teuthology.orchestra.run.smithi194.stdout:(5/23): librgw2-16.2.15-0.el8.x86_64.rpm 46 MB/s | 3.8 MB 00:00 2024-04-26T09:32:59.401 INFO:teuthology.orchestra.run.smithi194.stdout:(6/23): python3-ceph-argparse-16.2.15-0.el8.x86 1.8 MB/s | 45 kB 00:00 2024-04-26T09:32:59.435 INFO:teuthology.orchestra.run.smithi194.stdout:(7/23): python3-ceph-common-16.2.15-0.el8.x86_6 2.9 MB/s | 99 kB 00:00 2024-04-26T09:32:59.460 INFO:teuthology.orchestra.run.smithi194.stdout:(8/23): python3-cephfs-16.2.15-0.el8.x86_64.rpm 8.4 MB/s | 215 kB 00:00 2024-04-26T09:32:59.494 INFO:teuthology.orchestra.run.smithi194.stdout:(9/23): python3-rados-16.2.15-0.el8.x86_64.rpm 11 MB/s | 389 kB 00:00 2024-04-26T09:32:59.527 INFO:teuthology.orchestra.run.smithi194.stdout:(10/23): python3-rbd-16.2.15-0.el8.x86_64.rpm 11 MB/s | 369 kB 00:00 2024-04-26T09:32:59.553 INFO:teuthology.orchestra.run.smithi194.stdout:(11/23): python3-rgw-16.2.15-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-04-26T09:32:59.687 INFO:teuthology.orchestra.run.smithi194.stdout:(12/23): ceph-radosgw-16.2.15-0.el8.x86_64.rpm 18 MB/s | 12 MB 00:00 2024-04-26T09:32:59.953 INFO:teuthology.orchestra.run.smithi194.stdout:(13/23): ceph-common-16.2.15-0.el8.x86_64.rpm 27 MB/s | 24 MB 00:00 2024-04-26T09:33:00.470 INFO:teuthology.orchestra.run.smithi194.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 85 kB/s | 44 kB 00:00 2024-04-26T09:33:00.587 INFO:teuthology.orchestra.run.smithi194.stdout:(15/23): librabbitmq-0.9.0-5.el8.x86_64.rpm 405 kB/s | 47 kB 00:00 2024-04-26T09:33:00.796 INFO:teuthology.orchestra.run.smithi194.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 234 kB/s | 259 kB 00:01 2024-04-26T09:33:00.946 INFO:teuthology.orchestra.run.smithi194.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 854 kB/s | 306 kB 00:00 2024-04-26T09:33:01.038 INFO:teuthology.orchestra.run.smithi194.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 643 kB/s | 59 kB 00:00 2024-04-26T09:33:01.105 INFO:teuthology.orchestra.run.smithi194.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 586 kB/s | 181 kB 00:00 2024-04-26T09:33:01.138 INFO:teuthology.orchestra.run.smithi194.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 749 kB/s | 75 kB 00:00 2024-04-26T09:33:01.214 INFO:teuthology.orchestra.run.smithi194.stdout:(21/23): librados2-16.2.15-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2024-04-26T09:33:01.273 INFO:teuthology.orchestra.run.smithi194.stdout:(22/23): librbd1-16.2.15-0.el8.x86_64.rpm 31 MB/s | 4.2 MB 00:00 2024-04-26T09:33:05.764 INFO:teuthology.orchestra.run.smithi194.stdout:(23/23): librdkafka-1.6.1-1.el8.x86_64.rpm 109 kB/s | 677 kB 00:06 2024-04-26T09:33:05.766 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T09:33:05.766 INFO:teuthology.orchestra.run.smithi194.stdout:Total 8.0 MB/s | 58 MB 00:07 2024-04-26T09:33:05.810 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T09:33:05.857 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T09:33:05.857 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T09:33:06.200 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T09:33:06.203 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T09:33:07.178 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T09:33:07.386 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2024-04-26T09:33:07.564 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-26T09:33:07.950 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2024-04-26T09:33:08.131 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:16.2.15-0.el8.x86_64 3/25 2024-04-26T09:33:08.230 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.15-0.el8.x86_64 3/25 2024-04-26T09:33:08.504 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:16.2.15-0.el8.x86_64 4/25 2024-04-26T09:33:08.706 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:16.2.15-0.el8.x86_64 5/25 2024-04-26T09:33:09.129 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:16.2.15-0.el8.x86_64 5/25 2024-04-26T09:33:09.297 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:16.2.15-0.el8.x86_64 6/25 2024-04-26T09:33:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.15-0.el8.x86_64 6/25 2024-04-26T09:33:09.565 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/25 2024-04-26T09:33:10.035 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2024-04-26T09:33:10.189 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:16.2.15-0.el8.x86_64 9/25 2024-04-26T09:33:10.258 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:16.2.15-0.el8.x86_64 9/25 2024-04-26T09:33:10.444 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2024-04-26T09:33:10.602 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:16.2.15-0.el8.x86_64 11/25 2024-04-26T09:33:10.758 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:16.2.15-0.el8.x86_64 12/25 2024-04-26T09:33:10.956 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:16.2.15-0.el8.x86_64 13/25 2024-04-26T09:33:11.175 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:16.2.15-0.el8.x86_64 14/25 2024-04-26T09:33:11.338 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:16.2.15-0.el8.x86_64 15/25 2024-04-26T09:33:11.403 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:16.2.15-0.el8.x86_64 15/25 2024-04-26T09:33:11.614 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2024-04-26T09:33:11.805 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2024-04-26T09:33:11.983 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2024-04-26T09:33:12.189 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:16.2.15-0.el8.x86_64 19/25 2024-04-26T09:33:14.676 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:16.2.15-0.el8.x86_64 20/25 2024-04-26T09:33:14.897 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:16.2.15-0.el8.x86_64 20/25 2024-04-26T09:33:15.610 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:16.2.15-0.el8.x86_64 20/25 2024-04-26T09:33:15.930 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:16.2.15-0.el8.x86_64 21/25 2024-04-26T09:33:16.017 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:16.2.15-0.el8.x86_64 21/25 2024-04-26T09:33:16.138 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:16.2.15-0.el8.x86_64 22/25 2024-04-26T09:33:35.253 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:16.2.15-0.el8.x86_64 22/25 2024-04-26T09:33:35.253 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-04-26T09:33:35.253 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-04-26T09:33:35.253 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /dev 2024-04-26T09:33:35.253 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /run 2024-04-26T09:33:35.254 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-04-26T09:33:35.254 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-04-26T09:33:35.254 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-04-26T09:33:35.254 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-04-26T09:33:35.254 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /dev 2024-04-26T09:33:35.254 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:33:36.410 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:16.2.15-0.el8.x86_64 23/25 2024-04-26T09:33:36.468 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:16.2.15-0.el8.x86_64 23/25 2024-04-26T09:33:36.468 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-26T09:33:36.670 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2024-04-26T09:33:36.670 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-26T09:33:37.358 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2024-04-26T09:33:37.358 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:16.2.15-0.el8.x86_64 1/25 2024-04-26T09:33:37.358 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:16.2.15-0.el8.x86_64 2/25 2024-04-26T09:33:37.358 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:16.2.15-0.el8.x86_64 3/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:16.2.15-0.el8.x86_64 4/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:16.2.15-0.el8.x86_64 5/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:16.2.15-0.el8.x86_64 6/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:16.2.15-0.el8.x86_64 7/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:16.2.15-0.el8.x86_64 8/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:16.2.15-0.el8.x86_64 9/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:16.2.15-0.el8.x86_64 10/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:16.2.15-0.el8.x86_64 11/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:16.2.15-0.el8.x86_64 12/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:16.2.15-0.el8.x86_64 13/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2024-04-26T09:33:37.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2024-04-26T09:33:37.360 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 17/25 2024-04-26T09:33:37.360 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2024-04-26T09:33:37.360 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2024-04-26T09:33:37.360 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2024-04-26T09:33:37.360 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2024-04-26T09:33:37.360 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.15-0.el8.x86_64 22/25 2024-04-26T09:33:37.360 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2024-04-26T09:33:37.360 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.15-0.el8.x86_64 24/25 2024-04-26T09:33:38.213 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:16.2.15-0.el8.x86_64 librbd1-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: leveldb-1.22-1.el8.x86_64 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.2-3.el8.x86_64 2024-04-26T09:33:38.214 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.3.1-3.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:16.2.15-0.el8.x86_64 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:33:38.215 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:33:38.508 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test 2024-04-26T09:33:39.240 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:45 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:33:40.892 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:33:40.893 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:33:40.893 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-26T09:33:40.893 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:33:40.893 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T09:33:40.893 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:16.2.15-0.el8 ceph 52 M 2024-04-26T09:33:40.893 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-26T09:33:40.893 INFO:teuthology.orchestra.run.smithi194.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-04-26T09:33:40.893 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:16.2.15-0.el8 ceph 174 k 2024-04-26T09:33:40.893 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-04-26T09:33:40.893 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-04-26T09:33:40.894 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-04-26T09:33:40.894 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:33:40.894 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T09:33:40.894 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:33:40.894 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-04-26T09:33:40.894 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:33:40.895 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 53 M 2024-04-26T09:33:40.895 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 197 M 2024-04-26T09:33:40.895 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T09:33:41.225 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): libcephsqlite-16.2.15-0.el8.x86_64.rpm 1.2 MB/s | 174 kB 00:00 2024-04-26T09:33:41.809 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): ceph-test-16.2.15-0.el8.x86_64.rpm 72 MB/s | 52 MB 00:00 2024-04-26T09:33:43.176 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 97 kB/s | 203 kB 00:02 2024-04-26T09:33:43.376 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 87 kB/s | 188 kB 00:02 2024-04-26T09:33:44.492 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 54 kB/s | 71 kB 00:01 2024-04-26T09:33:47.967 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 52 kB/s | 323 kB 00:06 2024-04-26T09:33:47.968 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T09:33:47.969 INFO:teuthology.orchestra.run.smithi194.stdout:Total 7.5 MB/s | 53 MB 00:07 2024-04-26T09:33:48.011 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T09:33:48.031 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T09:33:48.031 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T09:33:48.354 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T09:33:48.357 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T09:33:48.834 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T09:33:49.034 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-04-26T09:33:49.235 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-04-26T09:33:49.366 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-26T09:33:49.446 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-04-26T09:33:49.613 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-04-26T09:33:49.765 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:16.2.15-0.el8.x86_64 5/6 2024-04-26T09:33:54.462 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:16.2.15-0.el8.x86_64 5/6 2024-04-26T09:33:54.696 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:16.2.15-0.el8.x86_64 6/6 2024-04-26T09:33:54.982 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-2:16.2.15-0.el8.x86_64 6/6 2024-04-26T09:33:54.982 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:16.2.15-0.el8.x86_64 1/6 2024-04-26T09:33:54.983 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:16.2.15-0.el8.x86_64 2/6 2024-04-26T09:33:54.983 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-04-26T09:33:54.983 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-04-26T09:33:54.983 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-04-26T09:33:55.398 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-04-26T09:33:55.399 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:33:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T09:33:55.399 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:16.2.15-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-04-26T09:33:55.399 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:16.2.15-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-04-26T09:33:55.399 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-04-26T09:33:55.399 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:33:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:33:55.598 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph 2024-04-26T09:33:56.338 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:02 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:33:57.989 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:33:57.992 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:33:57.992 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-04-26T09:33:57.992 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:33:57.992 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T09:33:57.992 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:16.2.15-0.el8 ceph 6.3 k 2024-04-26T09:33:57.992 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-26T09:33:57.992 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:16.2.15-0.el8 ceph 2.4 M 2024-04-26T09:33:57.992 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:16.2.15-0.el8 ceph 1.6 M 2024-04-26T09:33:57.992 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:16.2.15-0.el8 ceph-noarch 245 k 2024-04-26T09:33:57.992 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:16.2.15-0.el8 ceph 5.2 M 2024-04-26T09:33:57.993 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:16.2.15-0.el8 ceph 19 M 2024-04-26T09:33:57.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-04-26T09:33:57.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-04-26T09:33:57.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-04-26T09:33:57.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-04-26T09:33:57.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-04-26T09:33:57.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-04-26T09:33:57.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-04-26T09:33:57.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-04-26T09:33:57.993 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-04-26T09:33:57.994 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-04-26T09:33:57.995 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-04-26T09:33:57.995 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-04-26T09:33:57.995 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-04-26T09:33:57.995 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:33:57.995 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T09:33:57.995 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:33:57.995 INFO:teuthology.orchestra.run.smithi194.stdout:Install 32 Packages 2024-04-26T09:33:57.995 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:33:57.996 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 32 M 2024-04-26T09:33:57.996 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 104 M 2024-04-26T09:33:57.996 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T09:33:58.543 INFO:teuthology.orchestra.run.smithi194.stdout:(1/32): ceph-16.2.15-0.el8.x86_64.rpm 48 kB/s | 6.3 kB 00:00 2024-04-26T09:33:58.593 INFO:teuthology.orchestra.run.smithi194.stdout:(2/32): ceph-mgr-16.2.15-0.el8.x86_64.rpm 8.9 MB/s | 1.6 MB 00:00 2024-04-26T09:33:58.643 INFO:teuthology.orchestra.run.smithi194.stdout:(3/32): ceph-mds-16.2.15-0.el8.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-04-26T09:33:58.668 INFO:teuthology.orchestra.run.smithi194.stdout:(4/32): ceph-mgr-modules-core-16.2.15-0.el8.noa 9.7 MB/s | 245 kB 00:00 2024-04-26T09:33:58.753 INFO:teuthology.orchestra.run.smithi194.stdout:(5/32): ceph-mon-16.2.15-0.el8.x86_64.rpm 25 MB/s | 5.2 MB 00:00 2024-04-26T09:33:58.978 INFO:teuthology.orchestra.run.smithi194.stdout:(6/32): ceph-osd-16.2.15-0.el8.x86_64.rpm 49 MB/s | 19 MB 00:00 2024-04-26T09:33:59.003 INFO:teuthology.orchestra.run.smithi194.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 155 kB/s | 39 kB 00:00 2024-04-26T09:33:59.286 INFO:teuthology.orchestra.run.smithi194.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 201 kB/s | 57 kB 00:00 2024-04-26T09:33:59.453 INFO:teuthology.orchestra.run.smithi194.stdout:(9/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 218 kB/s | 103 kB 00:00 2024-04-26T09:33:59.737 INFO:teuthology.orchestra.run.smithi194.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 878 kB/s | 249 kB 00:00 2024-04-26T09:34:00.104 INFO:teuthology.orchestra.run.smithi194.stdout:(11/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-04-26T09:34:00.179 INFO:teuthology.orchestra.run.smithi194.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.4 MB/s | 109 kB 00:00 2024-04-26T09:34:00.379 INFO:teuthology.orchestra.run.smithi194.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 218 kB/s | 44 kB 00:00 2024-04-26T09:34:00.505 INFO:teuthology.orchestra.run.smithi194.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.4 MB/s | 185 kB 00:00 2024-04-26T09:34:00.580 INFO:teuthology.orchestra.run.smithi194.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.3 MB/s | 173 kB 00:00 2024-04-26T09:34:00.672 INFO:teuthology.orchestra.run.smithi194.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 4.1 MB/s | 384 kB 00:00 2024-04-26T09:34:00.722 INFO:teuthology.orchestra.run.smithi194.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 223 kB/s | 11 kB 00:00 2024-04-26T09:34:00.764 INFO:teuthology.orchestra.run.smithi194.stdout:(18/32): python3-mako-1.0.6-14.el8.noarch.rpm 75 kB/s | 157 kB 00:02 2024-04-26T09:34:00.789 INFO:teuthology.orchestra.run.smithi194.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 262 kB/s | 18 kB 00:00 2024-04-26T09:34:00.848 INFO:teuthology.orchestra.run.smithi194.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2024-04-26T09:34:00.915 INFO:teuthology.orchestra.run.smithi194.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 324 kB/s | 49 kB 00:00 2024-04-26T09:34:00.940 INFO:teuthology.orchestra.run.smithi194.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2024-04-26T09:34:00.974 INFO:teuthology.orchestra.run.smithi194.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 266 kB/s | 16 kB 00:00 2024-04-26T09:34:00.999 INFO:teuthology.orchestra.run.smithi194.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 318 kB/s | 19 kB 00:00 2024-04-26T09:34:01.032 INFO:teuthology.orchestra.run.smithi194.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 408 kB/s | 24 kB 00:00 2024-04-26T09:34:01.058 INFO:teuthology.orchestra.run.smithi194.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 500 kB/s | 29 kB 00:00 2024-04-26T09:34:01.100 INFO:teuthology.orchestra.run.smithi194.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 411 kB/s | 27 kB 00:00 2024-04-26T09:34:01.125 INFO:teuthology.orchestra.run.smithi194.stdout:(28/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.7 MB/s | 255 kB 00:00 2024-04-26T09:34:01.184 INFO:teuthology.orchestra.run.smithi194.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.4 MB/s | 85 kB 00:00 2024-04-26T09:34:01.252 INFO:teuthology.orchestra.run.smithi194.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 343 kB/s | 23 kB 00:00 2024-04-26T09:34:01.276 INFO:teuthology.orchestra.run.smithi194.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.4 MB/s | 251 kB 00:00 2024-04-26T09:34:02.276 INFO:teuthology.orchestra.run.smithi194.stdout:(32/32): python3-werkzeug-0.12.2-4.el8.noarch.r 153 kB/s | 457 kB 00:02 2024-04-26T09:34:02.277 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T09:34:02.278 INFO:teuthology.orchestra.run.smithi194.stdout:Total 7.4 MB/s | 32 MB 00:04 2024-04-26T09:34:02.358 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T09:34:02.396 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T09:34:02.396 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T09:34:02.710 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T09:34:02.714 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T09:34:03.795 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T09:34:04.018 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-04-26T09:34:04.213 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-04-26T09:34:04.412 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-04-26T09:34:04.653 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-04-26T09:34:04.851 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-04-26T09:34:05.040 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-04-26T09:34:05.258 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-04-26T09:34:05.452 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-04-26T09:34:05.686 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-04-26T09:34:05.940 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-04-26T09:34:06.152 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-04-26T09:34:06.385 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-04-26T09:34:06.750 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-04-26T09:34:06.998 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-04-26T09:34:07.204 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-04-26T09:34:07.444 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-04-26T09:34:07.646 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-26T09:34:07.953 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-04-26T09:34:08.160 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-04-26T09:34:08.331 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-04-26T09:34:08.509 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-04-26T09:34:08.751 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-04-26T09:34:08.962 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-04-26T09:34:09.180 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-04-26T09:34:09.495 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-04-26T09:34:09.798 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-04-26T09:34:10.165 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:16.2.15-0.el8.noarch 27/32 2024-04-26T09:34:10.747 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:16.2.15-0.el8.x86_64 28/32 2024-04-26T09:34:12.279 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:16.2.15-0.el8.x86_64 28/32 2024-04-26T09:34:12.445 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:16.2.15-0.el8.x86_64 29/32 2024-04-26T09:34:12.890 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:16.2.15-0.el8.x86_64 29/32 2024-04-26T09:34:13.045 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:16.2.15-0.el8.x86_64 30/32 2024-04-26T09:34:13.261 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:16.2.15-0.el8.x86_64 30/32 2024-04-26T09:34:13.429 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:16.2.15-0.el8.x86_64 31/32 2024-04-26T09:34:13.446 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:16.2.15-0.el8.x86_64 31/32 2024-04-26T09:34:13.578 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:16.2.15-0.el8.x86_64 32/32 2024-04-26T09:34:13.971 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-2:16.2.15-0.el8.x86_64 32/32 2024-04-26T09:34:13.971 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:16.2.15-0.el8.x86_64 1/32 2024-04-26T09:34:13.971 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:16.2.15-0.el8.x86_64 2/32 2024-04-26T09:34:13.971 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:16.2.15-0.el8.x86_64 3/32 2024-04-26T09:34:13.971 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:16.2.15-0.el8.x86_64 4/32 2024-04-26T09:34:13.971 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:16.2.15-0.el8.x86_64 5/32 2024-04-26T09:34:13.971 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:16.2.15-0.el8.noarch 6/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-04-26T09:34:13.972 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-04-26T09:34:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-04-26T09:34:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-04-26T09:34:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-04-26T09:34:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-04-26T09:34:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-04-26T09:34:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-04-26T09:34:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-04-26T09:34:13.973 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-04-26T09:34:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-04-26T09:34:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-04-26T09:34:13.974 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-04-26T09:34:14.736 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:16.2.15-0.el8.x86_64 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:16.2.15-0.el8.x86_64 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:16.2.15-0.el8.x86_64 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:16.2.15-0.el8.noarch 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:16.2.15-0.el8.x86_64 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:16.2.15-0.el8.x86_64 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-04-26T09:34:14.737 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-6.2-6.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.0.6-14.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-2.6-1.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.14-14.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2017.2-11.el8.noarch 2024-04-26T09:34:14.738 INFO:teuthology.orchestra.run.smithi194.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-04-26T09:34:14.739 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-04-26T09:34:14.739 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-04-26T09:34:14.739 INFO:teuthology.orchestra.run.smithi194.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-04-26T09:34:14.739 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-04-26T09:34:14.739 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-04-26T09:34:14.739 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-04-26T09:34:14.739 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-04-26T09:34:14.739 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-04-26T09:34:14.739 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:14.739 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:14.990 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base 2024-04-26T09:34:15.747 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:21 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:17.350 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-26T09:34:17.399 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:17.400 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-26T09:34:17.400 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:17.514 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-immutable-object-cache 2024-04-26T09:34:18.272 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:24 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:19.919 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:19.920 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:19.920 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-04-26T09:34:19.920 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:19.920 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T09:34:19.920 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:16.2.15-0.el8 ceph 161 k 2024-04-26T09:34:19.920 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:19.920 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T09:34:19.920 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:19.920 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-04-26T09:34:19.920 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:19.921 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 161 k 2024-04-26T09:34:19.921 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 470 k 2024-04-26T09:34:19.921 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T09:34:19.983 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-immutable-object-cache-16.2.15-0.el8.x86_6 2.7 MB/s | 161 kB 00:00 2024-04-26T09:34:19.983 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T09:34:19.983 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.7 MB/s | 161 kB 00:00 2024-04-26T09:34:19.983 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T09:34:19.990 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T09:34:19.990 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T09:34:20.039 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T09:34:20.042 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T09:34:20.149 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T09:34:20.302 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:20.699 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:21.083 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:21.083 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:21.083 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T09:34:21.083 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:16.2.15-0.el8.x86_64 2024-04-26T09:34:21.083 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:21.083 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:22.905 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr 2024-04-26T09:34:23.669 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:29 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:25.273 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-mgr-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-26T09:34:25.321 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:25.322 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-26T09:34:25.322 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:25.419 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse 2024-04-26T09:34:26.164 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:32 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:27.816 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:27.816 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:27.816 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-26T09:34:27.817 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:27.817 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T09:34:27.817 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:16.2.15-0.el8 ceph 911 k 2024-04-26T09:34:27.817 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-26T09:34:27.817 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.7-19.el8 baseos 83 k 2024-04-26T09:34:27.817 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:27.817 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T09:34:27.817 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:27.817 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-04-26T09:34:27.817 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:27.818 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 994 k 2024-04-26T09:34:27.818 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.8 M 2024-04-26T09:34:27.818 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T09:34:27.931 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): ceph-fuse-16.2.15-0.el8.x86_64.rpm 15 MB/s | 911 kB 00:00 2024-04-26T09:34:28.040 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): fuse-2.9.7-19.el8.x86_64.rpm 493 kB/s | 83 kB 00:00 2024-04-26T09:34:28.040 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T09:34:28.040 INFO:teuthology.orchestra.run.smithi194.stdout:Total 4.4 MB/s | 994 kB 00:00 2024-04-26T09:34:28.050 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T09:34:28.071 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T09:34:28.071 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T09:34:28.154 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T09:34:28.157 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T09:34:28.564 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T09:34:29.000 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.7-19.el8.x86_64 1/2 2024-04-26T09:34:29.167 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:16.2.15-0.el8.x86_64 2/2 2024-04-26T09:34:29.583 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-2:16.2.15-0.el8.x86_64 2/2 2024-04-26T09:34:29.583 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:16.2.15-0.el8.x86_64 1/2 2024-04-26T09:34:29.982 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.7-19.el8.x86_64 2/2 2024-04-26T09:34:29.983 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:29.983 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T09:34:29.983 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:16.2.15-0.el8.x86_64 fuse-2.9.7-19.el8.x86_64 2024-04-26T09:34:29.983 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:29.983 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:30.191 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel 2024-04-26T09:34:30.950 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:36 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:32.590 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:32.590 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:32.590 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-26T09:34:32.590 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:32.590 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T09:34:32.590 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:16.2.15-0.el8 ceph 131 k 2024-04-26T09:34:32.591 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:32.591 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T09:34:32.591 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:32.591 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-04-26T09:34:32.591 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:32.591 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 131 k 2024-04-26T09:34:32.591 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 432 k 2024-04-26T09:34:32.592 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T09:34:32.643 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-16.2.15-0.el8.x86_64.rpm 2.5 MB/s | 131 kB 00:00 2024-04-26T09:34:32.643 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T09:34:32.644 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.4 MB/s | 131 kB 00:00 2024-04-26T09:34:32.644 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T09:34:32.649 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T09:34:32.649 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T09:34:32.696 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T09:34:32.698 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T09:34:32.818 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T09:34:32.998 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:33.275 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:33.671 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:33.671 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:33.671 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T09:34:33.671 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:16.2.15-0.el8.x86_64 2024-04-26T09:34:33.672 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:33.672 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:33.893 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2 2024-04-26T09:34:34.649 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:40 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:36.263 INFO:teuthology.orchestra.run.smithi194.stdout:Package libcephfs2-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-26T09:34:36.310 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:36.311 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-26T09:34:36.311 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:36.412 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs-devel 2024-04-26T09:34:37.167 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:43 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:38.805 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:38.805 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-26T09:34:38.806 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:38.806 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T09:34:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:16.2.15-0.el8 ceph 25 k 2024-04-26T09:34:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:38.806 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T09:34:38.806 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:38.806 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-04-26T09:34:38.806 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:38.807 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 25 k 2024-04-26T09:34:38.807 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 107 k 2024-04-26T09:34:38.807 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T09:34:38.853 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs-devel-16.2.15-0.el8.x86_64.rpm 548 kB/s | 25 kB 00:00 2024-04-26T09:34:38.854 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T09:34:38.854 INFO:teuthology.orchestra.run.smithi194.stdout:Total 535 kB/s | 25 kB 00:00 2024-04-26T09:34:38.854 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T09:34:38.858 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T09:34:38.858 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T09:34:38.859 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T09:34:38.863 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T09:34:38.912 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T09:34:39.481 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:39.750 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:40.086 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:40.086 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:40.086 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T09:34:40.086 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:16.2.15-0.el8.x86_64 2024-04-26T09:34:40.087 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:40.087 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:40.209 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2 2024-04-26T09:34:40.963 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:46 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:42.569 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-26T09:34:42.618 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:42.619 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-26T09:34:42.619 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:42.726 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1 2024-04-26T09:34:43.482 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:49 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:45.090 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-26T09:34:45.139 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:45.140 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-26T09:34:45.140 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:45.422 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rados 2024-04-26T09:34:46.168 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:52 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:47.760 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rados-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-26T09:34:47.809 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:47.810 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-26T09:34:47.810 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:47.920 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rgw 2024-04-26T09:34:48.667 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:54 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:50.254 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rgw-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-26T09:34:50.302 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:50.303 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-26T09:34:50.303 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:50.436 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-cephfs 2024-04-26T09:34:51.188 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:57 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:52.807 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-cephfs-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-26T09:34:52.855 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:52.856 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-26T09:34:52.856 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:52.969 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rbd 2024-04-26T09:34:53.726 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:59 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:55.338 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rbd-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-26T09:34:55.388 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:55.389 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-26T09:34:55.389 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:55.611 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse 2024-04-26T09:34:56.367 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:02 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:34:58.013 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:34:58.014 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:58.014 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-26T09:34:58.014 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:58.014 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T09:34:58.014 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:16.2.15-0.el8 ceph 88 k 2024-04-26T09:34:58.014 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:58.014 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T09:34:58.014 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:34:58.014 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-04-26T09:34:58.015 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:58.015 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 88 k 2024-04-26T09:34:58.015 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 240 k 2024-04-26T09:34:58.015 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T09:34:58.066 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-16.2.15-0.el8.x86_64.rpm 1.7 MB/s | 88 kB 00:00 2024-04-26T09:34:58.066 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T09:34:58.066 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.7 MB/s | 88 kB 00:00 2024-04-26T09:34:58.067 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T09:34:58.076 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T09:34:58.076 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T09:34:58.123 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T09:34:58.126 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T09:34:58.224 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T09:34:58.394 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:58.691 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:34:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:59.094 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T09:34:59.094 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:16.2.15-0.el8.x86_64 2024-04-26T09:34:59.095 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:34:59.095 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:34:59.303 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror 2024-04-26T09:35:00.061 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:06 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:35:01.719 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:35:01.720 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:35:01.720 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-26T09:35:01.720 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:35:01.721 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T09:35:01.721 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:16.2.15-0.el8 ceph 3.7 M 2024-04-26T09:35:01.721 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:01.721 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T09:35:01.721 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:35:01.721 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-04-26T09:35:01.721 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:01.721 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 3.7 M 2024-04-26T09:35:01.721 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 15 M 2024-04-26T09:35:01.721 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T09:35:01.818 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-16.2.15-0.el8.x86_64.rpm 38 MB/s | 3.7 MB 00:00 2024-04-26T09:35:01.819 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T09:35:01.819 INFO:teuthology.orchestra.run.smithi194.stdout:Total 37 MB/s | 3.7 MB 00:00 2024-04-26T09:35:01.819 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T09:35:01.830 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T09:35:01.830 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T09:35:01.890 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T09:35:01.894 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T09:35:02.322 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T09:35:02.495 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:35:02.895 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:35:03.252 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:35:03.252 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:03.252 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T09:35:03.252 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:16.2.15-0.el8.x86_64 2024-04-26T09:35:03.252 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:03.252 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:35:03.469 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd 2024-04-26T09:35:04.235 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:10 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:35:05.886 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:35:05.887 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:35:05.887 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-04-26T09:35:05.887 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:35:05.887 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-26T09:35:05.887 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:16.2.15-0.el8 ceph 175 k 2024-04-26T09:35:05.887 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:05.887 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-26T09:35:05.887 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-04-26T09:35:05.887 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-04-26T09:35:05.888 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:05.888 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 175 k 2024-04-26T09:35:05.888 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 523 k 2024-04-26T09:35:05.888 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-26T09:35:05.936 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-16.2.15-0.el8.x86_64.rpm 3.5 MB/s | 175 kB 00:00 2024-04-26T09:35:05.937 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-26T09:35:05.937 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.4 MB/s | 175 kB 00:00 2024-04-26T09:35:05.937 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-26T09:35:05.948 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-26T09:35:05.948 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-26T09:35:05.994 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-26T09:35:05.997 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-26T09:35:06.107 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-26T09:35:06.301 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:35:06.591 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:35:06.967 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:16.2.15-0.el8.x86_64 1/1 2024-04-26T09:35:06.967 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:06.967 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-26T09:35:06.968 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:16.2.15-0.el8.x86_64 2024-04-26T09:35:06.968 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:06.968 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:35:07.150 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2 2024-04-26T09:35:07.908 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:13 ago on Fri 26 Apr 2024 09:32:54 AM UTC. 2024-04-26T09:35:09.516 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:16.2.15-0.el8.x86_64 is already installed. 2024-04-26T09:35:09.567 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:35:09.568 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-26T09:35:09.568 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:35:09.692 DEBUG:teuthology.parallel:result is None 2024-04-26T09:35:09.693 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T09:35:09.693 INFO:teuthology.packaging:ref: None 2024-04-26T09:35:09.693 INFO:teuthology.packaging:tag: None 2024-04-26T09:35:09.693 INFO:teuthology.packaging:branch: pacific 2024-04-26T09:35:09.693 INFO:teuthology.packaging:sha1: 19b26365dac54ee709df8c5dc2a7b7e89a50aba5 2024-04-26T09:35:09.693 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2024-04-26T09:35:09.922 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T09:35:09.944 INFO:teuthology.orchestra.run.smithi194.stdout:16.2.15-0.el8 2024-04-26T09:35:09.945 INFO:teuthology.packaging:The installed version of ceph is 16.2.15-0.el8 2024-04-26T09:35:09.945 INFO:teuthology.task.install:The correct ceph version 16.2.15-0 is installed. 2024-04-26T09:35:09.947 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T09:35:09.947 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:09.947 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T09:35:10.012 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T09:35:10.012 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:10.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T09:35:10.079 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T09:35:10.145 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T09:35:10.145 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:10.145 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T09:35:10.213 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T09:35:10.279 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T09:35:10.279 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:10.279 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T09:35:10.346 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T09:35:10.412 INFO:teuthology.run_tasks:Running task print... 2024-04-26T09:35:10.421 INFO:teuthology.task.print:**** done installing pacific 2024-04-26T09:35:10.421 INFO:teuthology.run_tasks:Running task ceph... 2024-04-26T09:35:10.503 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-26T09:35:10.503 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /var/log/ceph 2024-04-26T09:35:10.530 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-26T09:35:10.530 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T09:35:10.597 INFO:tasks.ceph:Creating extra log directories... 2024-04-26T09:35:10.597 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T09:35:10.672 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-26T09:35:10.672 INFO:tasks.ceph:config {'conf': {'global': {'mon warn on pool no app': False, 'ms bind msgr2': False, 'bluestore warn on legacy statfs': False, 'bluestore warn on no per pool omap': False, 'mon pg warn min per osd': 0}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on osd down out interval zero': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['overall HEALTH_', '\\(FS_', '\\(MDS_', '\\(OSD_', '\\(MON_DOWN\\)', '\\(CACHE_POOL_', '\\(POOL_', '\\(MGR_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'Monitor daemon marked osd', 'Behind on trimming', 'Manager daemon', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'scrub mismatch', 'ScrubResult', 'wrongly marked', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'overall HEALTH_', '\\(MON_MSGR2_NOT_ENABLED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-26T09:35:10.673 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639', 'branch': 'reef', 'description': 'fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7672639', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'install ceph/pacific latest\n'}], 'name': 'teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'bluestore warn on legacy statfs': False, 'bluestore warn on no per pool omap': False, 'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on osd down out interval zero': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'scrub mismatch', 'ScrubResult', 'wrongly marked', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'overall HEALTH_', '\\(MON_MSGR2_NOT_ENABLED\\)', 'slow request'], 'sha1': '19b26365dac54ee709df8c5dc2a7b7e89a50aba5'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '19b26365dac54ee709df8c5dc2a7b7e89a50aba5'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'dc093b5e41f728edab6e0330295fdf9156d31664'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 7931, 'sha1': '19b26365dac54ee709df8c5dc2a7b7e89a50aba5', 'sleep_before_teardown': 0, 'subset': '126/512', 'suite': 'fs', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_dc093b5e41f728edab6e0330295fdf9156d31664/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'dc093b5e41f728edab6e0330295fdf9156d31664', 'targets': {'smithi194.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAWlc2kZ/DO1NfCuDq1WvvsV03cE++vlE+/MHCAreg73mQ6TuDVK3Ctt9MmSJJpn8zoPeSeXJ6LxMHohlkrJac8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'branch': 'pacific', 'exclude_packages': ['librados3', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'cephadm', 'ceph-volume'], 'extra_packages': ['librados2'], 'flavor': 'default', 'sha1': '19b26365dac54ee709df8c5dc2a7b7e89a50aba5'}}, {'print': '**** done installing pacific'}, {'ceph': {'conf': {'global': {'mon warn on pool no app': False, 'ms bind msgr2': False, 'bluestore warn on legacy statfs': False, 'bluestore warn on no per pool omap': False, 'mon pg warn min per osd': 0}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on osd down out interval zero': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'log-ignorelist': ['overall HEALTH_', '\\(FS_', '\\(MDS_', '\\(OSD_', '\\(MON_DOWN\\)', '\\(CACHE_POOL_', '\\(POOL_', '\\(MGR_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'Monitor daemon marked osd', 'Behind on trimming', 'Manager daemon', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'scrub mismatch', 'ScrubResult', 'wrongly marked', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'overall HEALTH_', '\\(MON_MSGR2_NOT_ENABLED\\)', 'slow request'], 'flavor': 'default', 'fs': 'xfs', 'sha1': '19b26365dac54ee709df8c5dc2a7b7e89a50aba5', 'cluster': 'ceph'}}, {'exec': {'osd.0': ['ceph osd set-require-min-compat-client pacific']}}, {'print': '**** done ceph'}, {'print': '*** upgrading, no cephfs present'}, {'exec': {'mon.a': ['ceph fs dump']}}, {'install.upgrade': {'mon.a': None}}, {'print': '**** done install.upgrade'}, {'ceph.restart': {'daemons': ['mon.*', 'mgr.*'], 'mon-health-to-clog': False, 'wait-for-healthy': False}}, {'ceph.healthy': None}, {'ceph.restart': {'daemons': ['osd.*'], 'wait-for-healthy': False, 'wait-for-osds-up': True}}, {'exec': {'mon.a': ['ceph versions', 'ceph osd dump -f json-pretty', 'ceph fs dump', 'ceph osd require-osd-release quincy', 'for f in `ceph osd pool ls` ; do ceph osd pool set $f pg_autoscale_mode off ; done']}}, {'ceph.healthy': None}, {'print': '**** done ceph.restart'}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_22:24:15', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-26T09:35:10.673 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T09:35:10.722 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T09:35:10.789 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:10.789 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T09:35:10.847 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T09:35:10.847 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-04-26T09:35:10.904 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T09:35:10.905 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T09:35:10.905 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 197062 Links: 1 2024-04-26T09:35:10.905 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T09:35:10.905 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T09:35:10.905 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-26 09:35:02.650500597 +0000 2024-04-26T09:35:10.905 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-26 09:21:49.079462807 +0000 2024-04-26T09:35:10.905 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-26 09:21:49.079462807 +0000 2024-04-26T09:35:10.905 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-04-26T09:35:10.905 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T09:35:10.969 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-26T09:35:10.969 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-26T09:35:10.969 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000255325 s, 2.0 MB/s 2024-04-26T09:35:10.970 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T09:35:11.031 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-04-26T09:35:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T09:35:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T09:35:11.089 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 197167 Links: 1 2024-04-26T09:35:11.089 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T09:35:11.089 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T09:35:11.089 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-26 09:35:02.651500577 +0000 2024-04-26T09:35:11.089 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-26 09:21:49.408456149 +0000 2024-04-26T09:35:11.089 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-26 09:21:49.408456149 +0000 2024-04-26T09:35:11.089 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-04-26T09:35:11.090 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T09:35:11.155 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-26T09:35:11.155 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-26T09:35:11.155 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000242278 s, 2.1 MB/s 2024-04-26T09:35:11.157 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T09:35:11.215 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-04-26T09:35:11.273 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T09:35:11.273 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T09:35:11.273 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 197771 Links: 1 2024-04-26T09:35:11.273 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T09:35:11.273 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T09:35:11.273 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-26 09:35:02.651500577 +0000 2024-04-26T09:35:11.273 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-26 09:21:49.757449086 +0000 2024-04-26T09:35:11.273 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-26 09:21:49.757449086 +0000 2024-04-26T09:35:11.274 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-04-26T09:35:11.274 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T09:35:11.338 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-26T09:35:11.339 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-26T09:35:11.339 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000187938 s, 2.7 MB/s 2024-04-26T09:35:11.340 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T09:35:11.399 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-04-26T09:35:11.457 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T09:35:11.457 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T09:35:11.457 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 6h/6d Inode: 197283 Links: 1 2024-04-26T09:35:11.457 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T09:35:11.457 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-26T09:35:11.457 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-26 09:35:02.651500577 +0000 2024-04-26T09:35:11.457 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-26 09:21:50.082442509 +0000 2024-04-26T09:35:11.457 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-26 09:21:50.082442509 +0000 2024-04-26T09:35:11.457 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: - 2024-04-26T09:35:11.457 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T09:35:11.523 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-26T09:35:11.523 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-26T09:35:11.524 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.00020012 s, 2.6 MB/s 2024-04-26T09:35:11.524 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T09:35:11.583 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T09:35:11.583 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-04-26T09:35:11.583 INFO:tasks.ceph:Generating config... 2024-04-26T09:35:11.584 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-04-26T09:35:11.584 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-04-26T09:35:11.585 INFO:tasks.ceph:[global] bluestore warn on legacy statfs = False 2024-04-26T09:35:11.585 INFO:tasks.ceph:[global] bluestore warn on no per pool omap = False 2024-04-26T09:35:11.585 INFO:tasks.ceph:[global] mon pg warn min per osd = 0 2024-04-26T09:35:11.585 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-26T09:35:11.585 INFO:tasks.ceph:[client] debug client = 20 2024-04-26T09:35:11.585 INFO:tasks.ceph:[client] debug ms = 1 2024-04-26T09:35:11.585 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-26T09:35:11.585 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-26T09:35:11.585 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-26T09:35:11.585 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-26T09:35:11.585 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-26T09:35:11.585 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-26T09:35:11.585 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-26T09:35:11.585 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-26T09:35:11.585 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-26T09:35:11.586 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-26T09:35:11.586 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-26T09:35:11.586 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-26T09:35:11.586 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-26T09:35:11.586 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-26T09:35:11.586 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-26T09:35:11.586 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-26T09:35:11.586 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-26T09:35:11.586 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-26T09:35:11.586 INFO:tasks.ceph:[mon] mon warn on osd down out interval zero = False 2024-04-26T09:35:11.586 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-26T09:35:11.586 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-26T09:35:11.586 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-26T09:35:11.586 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-26T09:35:11.586 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-26T09:35:11.587 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-26T09:35:11.587 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-26T09:35:11.587 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-26T09:35:11.587 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-26T09:35:11.587 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-26T09:35:11.587 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-26T09:35:11.587 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-26T09:35:11.587 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-26T09:35:11.587 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-26T09:35:11.587 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-26T09:35:11.587 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-26T09:35:11.588 INFO:tasks.ceph:Setting up mon.a... 2024-04-26T09:35:11.588 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-26T09:35:11.734 INFO:teuthology.orchestra.run.smithi194.stdout:creating /etc/ceph/ceph.keyring 2024-04-26T09:35:11.737 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-26T09:35:11.777 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T09:35:11.844 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.194'), ('mon.b', '[v2:172.21.15.194:3301,v1:172.21.15.194:6790]'), ('mon.c', '[v2:172.21.15.194:3302,v1:172.21.15.194:6791]')] 2024-04-26T09:35:11.844 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.194,[v2:172.21.15.194:3301,v1:172.21.15.194:6790],[v2:172.21.15.194:3302,v1:172.21.15.194:6791]', 'mon warn on pool no app': False, 'ms bind msgr2': False, 'bluestore warn on legacy statfs': False, 'bluestore warn on no per pool omap': False, 'mon pg warn min per osd': 0}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120, 'mon warn on osd down out interval zero': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-26T09:35:11.845 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:11.846 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T09:35:11.902 DEBUG:teuthology.orchestra.run.smithi194:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.194 --addv b '[v2:172.21.15.194:3301,v1:172.21.15.194:6790]' --addv c '[v2:172.21.15.194:3302,v1:172.21.15.194:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-26T09:35:11.986 INFO:teuthology.orchestra.run.smithi194.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-26T09:35:11.986 INFO:teuthology.orchestra.run.smithi194.stdout:monmaptool: generated fsid be7f6752-916a-4f92-bdb5-9db81c891e5c 2024-04-26T09:35:11.987 INFO:teuthology.orchestra.run.smithi194.stdout:epoch 0 2024-04-26T09:35:11.987 INFO:teuthology.orchestra.run.smithi194.stdout:fsid be7f6752-916a-4f92-bdb5-9db81c891e5c 2024-04-26T09:35:11.987 INFO:teuthology.orchestra.run.smithi194.stdout:last_changed 2024-04-26T09:35:11.984130+0000 2024-04-26T09:35:11.987 INFO:teuthology.orchestra.run.smithi194.stdout:created 2024-04-26T09:35:11.984130+0000 2024-04-26T09:35:11.987 INFO:teuthology.orchestra.run.smithi194.stdout:min_mon_release 0 (unknown) 2024-04-26T09:35:11.987 INFO:teuthology.orchestra.run.smithi194.stdout:election_strategy: 1 2024-04-26T09:35:11.987 INFO:teuthology.orchestra.run.smithi194.stdout:0: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.a 2024-04-26T09:35:11.987 INFO:teuthology.orchestra.run.smithi194.stdout:1: [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] mon.b 2024-04-26T09:35:11.987 INFO:teuthology.orchestra.run.smithi194.stdout:2: [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] mon.c 2024-04-26T09:35:11.987 INFO:teuthology.orchestra.run.smithi194.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-26T09:35:11.988 DEBUG:teuthology.orchestra.run.smithi194:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T09:35:12.046 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID be7f6752-916a-4f92-bdb5-9db81c891e5c... 2024-04-26T09:35:12.047 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T09:35:12.130 INFO:teuthology.orchestra.run.smithi194.stdout:[global] 2024-04-26T09:35:12.130 INFO:teuthology.orchestra.run.smithi194.stdout: chdir = "" 2024-04-26T09:35:12.130 INFO:teuthology.orchestra.run.smithi194.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T09:35:12.130 INFO:teuthology.orchestra.run.smithi194.stdout: auth supported = cephx 2024-04-26T09:35:12.130 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: filestore xattr use omap = true 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: mon clock drift allowed = 1.000 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: osd crush chooseleaf type = 0 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: auth debug = true 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on old message = true 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on bug = true 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: mon pg warn max object skew = 0 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T09:35:12.131 INFO:teuthology.orchestra.run.smithi194.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default size = 2 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow primary affinity = true 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow pg remap = true 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on legacy crush tunables = false 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on crush straw calc version zero = false 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on no sortbitwise = false 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on osd down out interval zero = false 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on too few osds = false 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: mon_allow_pool_size_one = true 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T09:35:12.132 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: osd default data pool replay window = 5 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: mon allow pool delete = true 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: mon cluster log file level = debug 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: debug asserts on shutdown = true 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: mon health detail to clog = false 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: mon host = "172.21.15.194,[v2:172.21.15.194:3301,v1:172.21.15.194:6790],[v2:172.21.15.194:3302,v1:172.21.15.194:6791]" 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on pool no app = False 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: ms bind msgr2 = False 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore warn on legacy statfs = False 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore warn on no per pool omap = False 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: mon pg warn min per osd = 0 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: fsid = be7f6752-916a-4f92-bdb5-9db81c891e5c 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.133 INFO:teuthology.orchestra.run.smithi194.stdout:[osd] 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: osd journal size = 100 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub load threshold = 5.0 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub max interval = 600 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: osd mclock profile = high_recovery_ops 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: osd recover clone overlap = true 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: osd recovery max chunk = 1048576 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug shutdown = true 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug op order = true 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify stray on activate = true 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.134 INFO:teuthology.orchestra.run.smithi194.stdout: osd open classes on start = true 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug pg log writeout = true 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: osd deep scrub update digest min age = 30 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: osd map max advance = 10 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: journal zero on create = true 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: filestore ondisk finisher threads = 3 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: filestore apply finisher threads = 3 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: bdev debug aio = true 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug misdirected ops = true 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: bdev async discard = True 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: bdev enable discard = True 2024-04-26T09:35:12.135 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore allocator = bitmap 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore block size = 96636764160 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore fsck on mount = True 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluefs = 1/20 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluestore = 1/20 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: debug osd = 20 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: debug rocksdb = 4/10 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd full ratio = 0.9 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd nearfull ratio = 0.8 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: osd failsafe full ratio = 0.95 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: osd objectstore = bluestore 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: osd op complaint time = 180 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr] 2024-04-26T09:35:12.136 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: debug mgr = 20 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: mgr/telemetry/nag = false 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout:[mon] 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: debug paxos = 20 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: mon data avail warn = 5 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: mon mgr mkfs grace = 240 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-04-26T09:35:12.137 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd reporter subtree level = osd 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd prime pg temp = true 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: # 1m isn't quite enough 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: mon_down_mkfs_grace = 2m 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.138 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_filestore_osds = false 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: mon op complaint time = 120 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on osd down out interval zero = False 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout:[client] 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: rgw cache enabled = true 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable ops log = true 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable usage log = true 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: client mount timeout = 600 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: debug client = 20 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: rados mon op timeout = 900 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout: rados osd op timeout = 900 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.a] 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.b] 2024-04-26T09:35:12.139 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.c] 2024-04-26T09:35:12.140 INFO:teuthology.orchestra.run.smithi194.stdout:[mds] 2024-04-26T09:35:12.140 INFO:teuthology.orchestra.run.smithi194.stdout: debug mds = 20 2024-04-26T09:35:12.140 INFO:teuthology.orchestra.run.smithi194.stdout: debug mds balancer = 20 2024-04-26T09:35:12.140 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-04-26T09:35:12.140 INFO:teuthology.orchestra.run.smithi194.stdout: mds debug frag = True 2024-04-26T09:35:12.140 INFO:teuthology.orchestra.run.smithi194.stdout: mds debug scatterstat = True 2024-04-26T09:35:12.140 INFO:teuthology.orchestra.run.smithi194.stdout: mds op complaint time = 180 2024-04-26T09:35:12.140 INFO:teuthology.orchestra.run.smithi194.stdout: mds verify scatter = True 2024-04-26T09:35:12.140 INFO:teuthology.orchestra.run.smithi194.stdout: osd op complaint time = 180 2024-04-26T09:35:12.140 INFO:teuthology.orchestra.run.smithi194.stdout: rados mon op timeout = 900 2024-04-26T09:35:12.140 INFO:teuthology.orchestra.run.smithi194.stdout: rados osd op timeout = 900 2024-04-26T09:35:12.141 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-26T09:35:12.141 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-26T09:35:12.233 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-26T09:35:12.233 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:12.233 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-26T09:35:12.251 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:12.251 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-26T09:35:12.308 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:35:12.308 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:12.308 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T09:35:12.308 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T09:35:12.384 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:12.384 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T09:35:12.442 INFO:tasks.ceph:Setting up mon nodes... 2024-04-26T09:35:12.442 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-26T09:35:12.442 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T09:35:12.531 INFO:teuthology.orchestra.run.smithi194.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T09:35:12.533 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-26T09:35:12.581 INFO:teuthology.orchestra.run.smithi194.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-26T09:35:12.583 INFO:tasks.ceph:Setting up mds nodes... 2024-04-26T09:35:12.583 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-26T09:35:12.583 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-26T09:35:12.584 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-04-26T09:35:12.584 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-26T09:35:12.609 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T09:35:12.610 INFO:tasks.ceph:role: osd.0 2024-04-26T09:35:12.610 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:35:12.610 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T09:35:12.684 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T09:35:12.684 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T09:35:12.684 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T09:35:12.684 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T09:35:12.684 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T09:35:12.684 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-04-26T09:35:12.684 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T09:35:12.684 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T09:35:12.684 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T09:35:12.684 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T09:35:13.119 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-04-26T09:35:13.121 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-04-26T09:35:13.121 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-26T09:35:13.299 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-26T09:35:13.332 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-26T09:35:13.402 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T09:35:13.402 INFO:tasks.ceph:role: osd.1 2024-04-26T09:35:13.403 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:35:13.403 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T09:35:13.477 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T09:35:13.477 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T09:35:13.477 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T09:35:13.477 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T09:35:13.478 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T09:35:13.478 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-04-26T09:35:13.478 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T09:35:13.478 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T09:35:13.478 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T09:35:13.478 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T09:35:13.902 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-04-26T09:35:13.904 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-04-26T09:35:13.904 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-26T09:35:14.012 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-26T09:35:14.044 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-26T09:35:14.112 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T09:35:14.112 INFO:tasks.ceph:role: osd.2 2024-04-26T09:35:14.112 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:35:14.113 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T09:35:14.186 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T09:35:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T09:35:14.186 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T09:35:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T09:35:14.187 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T09:35:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-04-26T09:35:14.187 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T09:35:14.187 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T09:35:14.187 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T09:35:14.187 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T09:35:14.638 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-04-26T09:35:14.640 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-04-26T09:35:14.640 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-26T09:35:14.810 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-26T09:35:14.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-26T09:35:14.910 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-26T09:35:14.910 INFO:tasks.ceph:role: osd.3 2024-04-26T09:35:14.910 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:35:14.911 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-26T09:35:14.985 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T09:35:14.985 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T09:35:14.985 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T09:35:14.985 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T09:35:14.985 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T09:35:14.986 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-04-26T09:35:14.986 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T09:35:14.986 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T09:35:14.986 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T09:35:14.986 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T09:35:15.426 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-04-26T09:35:15.427 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-04-26T09:35:15.428 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-26T09:35:15.591 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-26T09:35:15.625 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T09:35:15.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:15.714+0000 7efe08016380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-26T09:35:15.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:15.715+0000 7efe08016380 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-26T09:35:15.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:15.715+0000 7efe08016380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T09:35:15.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:15.715+0000 7efe08016380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T09:35:15.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:15.715+0000 7efe08016380 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T09:35:15.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:15.715+0000 7efe08016380 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-26T09:35:18.563 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-26T09:35:18.595 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T09:35:18.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:18.679+0000 7fb2a51cb380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-26T09:35:18.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:18.680+0000 7fb2a51cb380 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-26T09:35:18.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:18.680+0000 7fb2a51cb380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T09:35:18.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:18.680+0000 7fb2a51cb380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T09:35:18.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:18.680+0000 7fb2a51cb380 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T09:35:18.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:18.680+0000 7fb2a51cb380 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-26T09:35:21.523 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-26T09:35:21.552 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T09:35:21.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:21.633+0000 7f23a9bea380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-26T09:35:21.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:21.633+0000 7f23a9bea380 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-26T09:35:21.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:21.633+0000 7f23a9bea380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T09:35:21.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:21.633+0000 7f23a9bea380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T09:35:21.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:21.633+0000 7f23a9bea380 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T09:35:21.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:21.634+0000 7f23a9bea380 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-26T09:35:24.479 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-26T09:35:24.549 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T09:35:24.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:24.632+0000 7fb286629380 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-26T09:35:24.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:24.633+0000 7fb286629380 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-26T09:35:24.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:24.633+0000 7fb286629380 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T09:35:24.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:24.633+0000 7fb286629380 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T09:35:24.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:24.633+0000 7fb286629380 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T09:35:24.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:24.633+0000 7fb286629380 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-26T09:35:27.477 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-26T09:35:27.508 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-26T09:35:27.509 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:27.509 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-26T09:35:27.580 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:27.581 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-26T09:35:27.651 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:27.651 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-26T09:35:27.720 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:27.720 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-26T09:35:27.789 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:27.789 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-26T09:35:27.857 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:27.857 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-26T09:35:27.926 INFO:tasks.ceph:Adding keys to all mons... 2024-04-26T09:35:27.926 DEBUG:teuthology.orchestra.run.smithi194:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T09:35:27.992 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.x] 2024-04-26T09:35:27.992 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBQdStm8uSFHxAAv9eg6uxUdq5ZzX2uMWdBCw== 2024-04-26T09:35:27.993 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.y] 2024-04-26T09:35:27.993 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBQdStmidyHIhAA42X/37OntmZgd+hq24Fa/g== 2024-04-26T09:35:27.993 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.0] 2024-04-26T09:35:27.993 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBTdStmFLifKhAAfRxDMH9FliKQ7UCwonFJzw== 2024-04-26T09:35:27.993 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.1] 2024-04-26T09:35:27.993 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBWdStmrGmKKBAAv7W7LjRifD754fuEWRgiBw== 2024-04-26T09:35:27.993 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.2] 2024-04-26T09:35:27.993 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBZdStmYYLFJRAA/irifonPdvuPPW9bLOD21w== 2024-04-26T09:35:27.993 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.3] 2024-04-26T09:35:27.993 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBcdStmVVq8JRAAJioMHSmTFY2jdEna0a3OhQ== 2024-04-26T09:35:27.994 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T09:35:28.083 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T09:35:28.131 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T09:35:28.178 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T09:35:28.227 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T09:35:28.275 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T09:35:28.323 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-26T09:35:28.323 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-26T09:35:28.349 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T09:35:28.443 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-26T09:35:28.469 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-26T09:35:28.536 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T09:35:28.629 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-26T09:35:28.656 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-26T09:35:28.723 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T09:35:28.816 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-26T09:35:28.842 DEBUG:teuthology.orchestra.run.smithi194:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T09:35:28.900 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-26T09:35:28.900 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T09:35:28.900 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T09:35:28.942 INFO:tasks.ceph.mon.a:Started 2024-04-26T09:35:28.943 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T09:35:28.943 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T09:35:28.944 INFO:tasks.ceph.mon.b:Started 2024-04-26T09:35:28.945 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T09:35:28.945 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T09:35:28.946 INFO:tasks.ceph.mon.c:Started 2024-04-26T09:35:28.946 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-26T09:35:28.947 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-26T09:35:28.947 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-26T09:35:28.949 INFO:tasks.ceph.mgr.x:Started 2024-04-26T09:35:28.949 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-26T09:35:28.949 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-26T09:35:28.951 INFO:tasks.ceph.mgr.y:Started 2024-04-26T09:35:28.951 INFO:tasks.ceph:Setting crush tunables to default 2024-04-26T09:35:28.951 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd crush tunables default 2024-04-26T09:35:29.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.093+0000 7efd9c098700 1 Processor -- start 2024-04-26T09:35:29.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.093+0000 7efd9c098700 1 -- start start 2024-04-26T09:35:29.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd9c098700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 0x7efd94090430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd9c098700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7efd9412ac60 0x7efd9412d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd9c098700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9408ef10 con 0x7efd94090920 2024-04-26T09:35:29.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd9c098700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd94069e10 con 0x7efd9412ac60 2024-04-26T09:35:29.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd9c098700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7efd940654a0 con 0x7efd9405e230 2024-04-26T09:35:29.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd99e34700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 0x7efd94090430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd99e34700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 0x7efd94090430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46836/0 (socket says 172.21.15.194:46836) 2024-04-26T09:35:29.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd99e34700 1 -- 172.21.15.194:0/1112182888 learned_addr learned my addr 172.21.15.194:0/1112182888 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:29.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd99633700 1 --1- 172.21.15.194:0/1112182888 >> v1:172.21.15.194:6789/0 conn(0x7efd94090920 0x7efd94090d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57002/0 (socket says 172.21.15.194:57002) 2024-04-26T09:35:29.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd9a635700 1 --2- 172.21.15.194:0/1112182888 >> v2:172.21.15.194:3300/0 conn(0x7efd9412ac60 0x7efd9412d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd99e34700 1 -- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 msgr2=0x7efd94090430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T09:35:29.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd99e34700 1 -- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 msgr2=0x7efd94090430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T09:35:29.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd99e34700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 0x7efd94090430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-26T09:35:29.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd9a635700 1 -- 172.21.15.194:0/1112182888 >> v2:172.21.15.194:3300/0 conn(0x7efd9412ac60 msgr2=0x7efd9412d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T09:35:29.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd9a635700 1 -- 172.21.15.194:0/1112182888 >> v2:172.21.15.194:3300/0 conn(0x7efd9412ac60 msgr2=0x7efd9412d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T09:35:29.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd9a635700 1 --2- 172.21.15.194:0/1112182888 >> v2:172.21.15.194:3300/0 conn(0x7efd9412ac60 0x7efd9412d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-04-26T09:35:29.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd99e34700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 0x7efd94090430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-04-26T09:35:29.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.094+0000 7efd9a635700 1 --2- 172.21.15.194:0/1112182888 >> v2:172.21.15.194:3300/0 conn(0x7efd9412ac60 0x7efd9412d050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2024-04-26T09:35:29.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.129+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596160212 0 0) 0x7efd9408ef10 con 0x7efd94090920 2024-04-26T09:35:29.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd800033f0 con 0x7efd94090920 2024-04-26T09:35:29.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 <== mon.0 v1:172.21.15.194:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 147698285 0 0) 0x7efd800033f0 con 0x7efd94090920 2024-04-26T09:35:29.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 msgr2=0x7efd94090430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T09:35:29.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 0x7efd94090430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 >> v2:172.21.15.194:3300/0 conn(0x7efd9412ac60 msgr2=0x7efd9412d050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T09:35:29.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 --2- 172.21.15.194:0/1112182888 >> v2:172.21.15.194:3300/0 conn(0x7efd9412ac60 0x7efd9412d050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 --> v1:172.21.15.194:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd940667b0 con 0x7efd94090920 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 <== mon.0 v1:172.21.15.194:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2670628180 0 0) 0x7efd84001470 con 0x7efd94090920 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 >> v1:172.21.15.194:6789/0 conn(0x7efd94090920 legacy=0x7efd94090d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 0x7efd80003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 0x7efd80004690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd80004b80 0x7efd80005000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd94069e10 con 0x7efd80004b80 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd940654a0 con 0x7efd80003ef0 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7efd8000b990 con 0x7efd9405e230 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd99e34700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 0x7efd80003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd99633700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 0x7efd80004690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd9a635700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd80004b80 0x7efd80005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd99633700 1 -- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 msgr2=0x7efd80003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd99633700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 0x7efd80003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd99633700 1 -- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd80004b80 msgr2=0x7efd80005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:29.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd99633700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd80004b80 0x7efd80005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd99633700 1 -- 172.21.15.194:0/1112182888 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd800033f0 con 0x7efd80003ef0 2024-04-26T09:35:29.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd99e34700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9405e230 0x7efd80003910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:29.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd9a635700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd80004b80 0x7efd80005000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:29.137 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.130+0000 7efd99633700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 0x7efd80004690 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7efd940650f0 tx=0x7efd94066940).ready entity=mon.1 client_cookie=220c525fa5b75454 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:29.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.133+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8400c980 con 0x7efd80003ef0 2024-04-26T09:35:29.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.133+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8400cae0 con 0x7efd80003ef0 2024-04-26T09:35:29.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.133+0000 7efd98e32700 1 -- 172.21.15.194:0/1112182888 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8400ce60 con 0x7efd80003ef0 2024-04-26T09:35:29.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.133+0000 7efd9c098700 1 -- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 msgr2=0x7efd80004690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.133+0000 7efd9c098700 1 --2- 172.21.15.194:0/1112182888 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 0x7efd80004690 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7efd940650f0 tx=0x7efd94066940).stop 2024-04-26T09:35:29.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.133+0000 7efd9a635700 1 -- 172.21.15.194:0/1112182888 reap_dead start 2024-04-26T09:35:29.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.133+0000 7efd9c098700 1 -- 172.21.15.194:0/1112182888 shutdown_connections 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.133+0000 7efd9c098700 1 -- 172.21.15.194:0/1112182888 >> 172.21.15.194:0/1112182888 conn(0x7efd94092180 msgr2=0x7efd94094a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.133+0000 7efd9c098700 1 -- 172.21.15.194:0/1112182888 shutdown_connections 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.133+0000 7efd9c098700 1 -- 172.21.15.194:0/1112182888 wait complete. 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.134+0000 7efd9c098700 1 Processor -- start 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.134+0000 7efd9c098700 1 -- start start 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd9c098700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 0x7efd9411a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd9c098700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd9411a730 0x7efd9411abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd9c098700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9411b0a0 0x7efd9413b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd9c098700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9412a8a0 con 0x7efd9411a730 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd9c098700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9411b500 con 0x7efd80003ef0 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd9c098700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7efd9411d760 con 0x7efd9411b0a0 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd99e34700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 0x7efd9411a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd9a635700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9411b0a0 0x7efd9413b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd99633700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd9411a730 0x7efd9411abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd99e34700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 0x7efd9411a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:36636/0 (socket says 172.21.15.194:36636) 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd99e34700 1 -- 172.21.15.194:0/3493085427 learned_addr learned my addr 172.21.15.194:0/3493085427 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd99e34700 1 -- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9411b0a0 msgr2=0x7efd9413b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd99e34700 1 --2- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9411b0a0 0x7efd9413b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd99e34700 1 -- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd9411a730 msgr2=0x7efd9411abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd99e34700 1 --2- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd9411a730 0x7efd9411abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd99e34700 1 -- 172.21.15.194:0/3493085427 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9411db30 con 0x7efd80003ef0 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd99633700 1 --2- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd9411a730 0x7efd9411abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd9a635700 1 --2- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9411b0a0 0x7efd9413b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd99e34700 1 --2- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 0x7efd9411a240 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7efd90005fd0 tx=0x7efd90008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd8a7fc700 1 -- 172.21.15.194:0/3493085427 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9000b3f0 con 0x7efd80003ef0 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd8a7fc700 1 -- 172.21.15.194:0/3493085427 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90010740 con 0x7efd80003ef0 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd9c098700 1 -- 172.21.15.194:0/3493085427 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9411d0a0 con 0x7efd80003ef0 2024-04-26T09:35:29.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd8a7fc700 1 -- 172.21.15.194:0/3493085427 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90010ac0 con 0x7efd80003ef0 2024-04-26T09:35:29.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.135+0000 7efd9c098700 1 -- 172.21.15.194:0/3493085427 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9413b600 con 0x7efd80003ef0 2024-04-26T09:35:29.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.136+0000 7efd8a7fc700 1 -- 172.21.15.194:0/3493085427 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7efd900146d0 con 0x7efd80003ef0 2024-04-26T09:35:29.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.136+0000 7efd8a7fc700 1 -- 172.21.15.194:0/3493085427 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7efd9001e7b0 con 0x7efd80003ef0 2024-04-26T09:35:29.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.136+0000 7efd9c098700 1 -- 172.21.15.194:0/3493085427 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd940906f0 con 0x7efd80003ef0 2024-04-26T09:35:29.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.139+0000 7efd8a7fc700 1 -- 172.21.15.194:0/3493085427 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7efd90014b60 con 0x7efd80003ef0 2024-04-26T09:35:29.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.208+0000 7efd9c098700 1 -- 172.21.15.194:0/3493085427 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7efd94095150 con 0x7efd80003ef0 2024-04-26T09:35:29.219 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.216+0000 7efd8a7fc700 1 -- 172.21.15.194:0/3493085427 <== mon.1 v2:172.21.15.194:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7efd90014d00 con 0x7efd80003ef0 2024-04-26T09:35:29.220 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.216+0000 7efd9c098700 1 -- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 msgr2=0x7efd9411a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.220 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.216+0000 7efd9c098700 1 --2- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 0x7efd9411a240 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7efd90005fd0 tx=0x7efd90008040).stop 2024-04-26T09:35:29.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.217+0000 7efd9c098700 1 -- 172.21.15.194:0/3493085427 shutdown_connections 2024-04-26T09:35:29.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.217+0000 7efd9c098700 1 --2- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efd9411b0a0 0x7efd9413b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.217+0000 7efd9c098700 1 --2- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efd80003ef0 0x7efd9411a240 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.217+0000 7efd9c098700 1 --2- 172.21.15.194:0/3493085427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd9411a730 0x7efd9411abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.217+0000 7efd9c098700 1 -- 172.21.15.194:0/3493085427 >> 172.21.15.194:0/3493085427 conn(0x7efd94092180 msgr2=0x7efd94068970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:29.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.217+0000 7efd9c098700 1 -- 172.21.15.194:0/3493085427 shutdown_connections 2024-04-26T09:35:29.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.217+0000 7efd9c098700 1 -- 172.21.15.194:0/3493085427 wait complete. 2024-04-26T09:35:29.221 INFO:teuthology.orchestra.run.smithi194.stderr:adjusted tunables profile to default 2024-04-26T09:35:29.231 INFO:tasks.ceph:check_enable_crimson: False 2024-04-26T09:35:29.231 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-26T09:35:29.232 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:29.232 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-26T09:35:29.261 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:29.261 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-26T09:35:29.329 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:29.329 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-26T09:35:29.372 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:29.368+0000 7fdf4e007680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T09:35:29.372 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:29.368+0000 7fc59c9f1680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T09:35:29.395 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:29.395 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-26T09:35:29.444 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:29.440+0000 7fc59c9f1680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T09:35:29.445 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:29.442+0000 7fdf4e007680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T09:35:29.463 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 79a6c3fb-32e2-46b3-81f7-7ab1e3bf3e5a 0 2024-04-26T09:35:29.519 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:29.516+0000 7fdf4e007680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T09:35:29.521 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:29.517+0000 7fc59c9f1680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T09:35:29.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.626+0000 7f8f6cf5c700 1 Processor -- start 2024-04-26T09:35:29.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.626+0000 7f8f6cf5c700 1 -- start start 2024-04-26T09:35:29.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.626+0000 7f8f6cf5c700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68066e70 0x7f8f68067250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.626+0000 7f8f6cf5c700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f8f68067740 0x7f8f6811c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.626+0000 7f8f6cf5c700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f680688f0 con 0x7f8f6811c5c0 2024-04-26T09:35:29.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.626+0000 7f8f6cf5c700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f6811d180 con 0x7f8f68067740 2024-04-26T09:35:29.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.626+0000 7f8f6cf5c700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f68065090 con 0x7f8f68066e70 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.626+0000 7f8f66d9d700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f8f6811c5c0 0x7f8f6812ce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57020/0 (socket says 172.21.15.194:57020) 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.626+0000 7f8f66d9d700 1 -- 172.21.15.194:0/3856070608 learned_addr learned my addr 172.21.15.194:0/3856070608 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.626+0000 7f8f6659c700 1 --2- 172.21.15.194:0/3856070608 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68066e70 0x7f8f68067250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.627+0000 7f8f65d9b700 1 --2- 172.21.15.194:0/3856070608 >> v2:172.21.15.194:3300/0 conn(0x7f8f68067740 0x7f8f6811c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.627+0000 7f8f6659c700 1 -- 172.21.15.194:0/3856070608 >> v2:172.21.15.194:3300/0 conn(0x7f8f68067740 msgr2=0x7f8f6811c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.627+0000 7f8f6659c700 1 --2- 172.21.15.194:0/3856070608 >> v2:172.21.15.194:3300/0 conn(0x7f8f68067740 0x7f8f6811c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.627+0000 7f8f6659c700 1 -- 172.21.15.194:0/3856070608 >> v1:172.21.15.194:6789/0 conn(0x7f8f6811c5c0 legacy=0x7f8f6812ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.627+0000 7f8f6559a700 1 -- 172.21.15.194:0/3856070608 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647645209 0 0) 0x7f8f680688f0 con 0x7f8f6811c5c0 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.627+0000 7f8f6659c700 1 -- 172.21.15.194:0/3856070608 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f68068bb0 con 0x7f8f68066e70 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.627+0000 7f8f65d9b700 1 --2- 172.21.15.194:0/3856070608 >> v2:172.21.15.194:3300/0 conn(0x7f8f68067740 0x7f8f6811c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.627+0000 7f8f6659c700 1 --2- 172.21.15.194:0/3856070608 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68066e70 0x7f8f68067250 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f8f58006fd0 tx=0x7f8f58006f40).ready entity=mon.1 client_cookie=f79c80f2a93d6f5e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.627+0000 7f8f6559a700 1 -- 172.21.15.194:0/3856070608 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f58005880 con 0x7f8f68066e70 2024-04-26T09:35:29.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.627+0000 7f8f6559a700 1 -- 172.21.15.194:0/3856070608 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f580059e0 con 0x7f8f68066e70 2024-04-26T09:35:29.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.627+0000 7f8f6559a700 1 -- 172.21.15.194:0/3856070608 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f58005d60 con 0x7f8f68066e70 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.628+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3856070608 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68066e70 msgr2=0x7f8f68067250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.628+0000 7f8f6cf5c700 1 --2- 172.21.15.194:0/3856070608 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68066e70 0x7f8f68067250 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f8f58006fd0 tx=0x7f8f58006f40).stop 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.628+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3856070608 shutdown_connections 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.628+0000 7f8f6cf5c700 1 --2- 172.21.15.194:0/3856070608 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68066e70 0x7f8f68067250 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.628+0000 7f8f6cf5c700 1 --2- 172.21.15.194:0/3856070608 >> v2:172.21.15.194:3300/0 conn(0x7f8f68067740 0x7f8f6811c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.628+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3856070608 >> 172.21.15.194:0/3856070608 conn(0x7f8f68091bc0 msgr2=0x7f8f68091fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.628+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3856070608 shutdown_connections 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.628+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3856070608 wait complete. 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.629+0000 7f8f6cf5c700 1 Processor -- start 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.629+0000 7f8f6cf5c700 1 -- start start 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.629+0000 7f8f6cf5c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8f68066e70 0x7f8f681352f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.629+0000 7f8f6cf5c700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68067740 0x7f8f681357e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.629+0000 7f8f6cf5c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f6811c5c0 0x7f8f68123880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.629+0000 7f8f6cf5c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f681108c0 con 0x7f8f6811c5c0 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.629+0000 7f8f6cf5c700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f68125400 con 0x7f8f68067740 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.629+0000 7f8f6cf5c700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8f68126520 con 0x7f8f68066e70 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f6659c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8f68066e70 0x7f8f681352f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f6659c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8f68066e70 0x7f8f681352f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46896/0 (socket says 172.21.15.194:46896) 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f65d9b700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68067740 0x7f8f681357e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f6659c700 1 -- 172.21.15.194:0/3931093037 learned_addr learned my addr 172.21.15.194:0/3931093037 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f66d9d700 1 --2- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f6811c5c0 0x7f8f68123880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f6659c700 1 -- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68067740 msgr2=0x7f8f681357e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f6659c700 1 --2- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68067740 0x7f8f681357e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f6659c700 1 -- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f6811c5c0 msgr2=0x7f8f68123880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f6659c700 1 --2- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f6811c5c0 0x7f8f68123880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f6659c700 1 -- 172.21.15.194:0/3931093037 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c0092a0 con 0x7f8f68066e70 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f65d9b700 1 --2- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68067740 0x7f8f681357e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f66d9d700 1 --2- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f6811c5c0 0x7f8f68123880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:29.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f6659c700 1 --2- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8f68066e70 0x7f8f681352f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8f58006f80 tx=0x7f8f58016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:29.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f56ffd700 1 -- 172.21.15.194:0/3931093037 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5800f720 con 0x7f8f68066e70 2024-04-26T09:35:29.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f56ffd700 1 -- 172.21.15.194:0/3931093037 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5800f880 con 0x7f8f68066e70 2024-04-26T09:35:29.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3931093037 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f58009250 con 0x7f8f68066e70 2024-04-26T09:35:29.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f56ffd700 1 -- 172.21.15.194:0/3931093037 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5800fc40 con 0x7f8f68066e70 2024-04-26T09:35:29.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.630+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3931093037 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f68135cd0 con 0x7f8f68066e70 2024-04-26T09:35:29.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.631+0000 7f8f56ffd700 1 -- 172.21.15.194:0/3931093037 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f8f58011650 con 0x7f8f68066e70 2024-04-26T09:35:29.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.631+0000 7f8f56ffd700 1 -- 172.21.15.194:0/3931093037 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8f58026790 con 0x7f8f68066e70 2024-04-26T09:35:29.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.632+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3931093037 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f681269f0 con 0x7f8f68066e70 2024-04-26T09:35:29.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.636+0000 7f8f56ffd700 1 -- 172.21.15.194:0/3931093037 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f8f681269f0 con 0x7f8f68066e70 2024-04-26T09:35:29.676 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:29.673+0000 7fc59c9f1680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T09:35:29.677 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:29.674+0000 7fdf4e007680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T09:35:29.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.717+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3931093037 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "osd new", "uuid": "79a6c3fb-32e2-46b3-81f7-7ab1e3bf3e5a", "id": 0} v 0) v1 -- 0x7f8f68110380 con 0x7f8f68066e70 2024-04-26T09:35:29.727 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.723+0000 7f8f56ffd700 1 -- 172.21.15.194:0/3931093037 <== mon.2 v2:172.21.15.194:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "79a6c3fb-32e2-46b3-81f7-7ab1e3bf3e5a", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8f5802b060 con 0x7f8f68066e70 2024-04-26T09:35:29.727 INFO:teuthology.orchestra.run.smithi194.stdout:0 2024-04-26T09:35:29.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.724+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8f68066e70 msgr2=0x7f8f681352f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.724+0000 7f8f6cf5c700 1 --2- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8f68066e70 0x7f8f681352f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8f58006f80 tx=0x7f8f58016000).stop 2024-04-26T09:35:29.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.724+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3931093037 shutdown_connections 2024-04-26T09:35:29.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.724+0000 7f8f6cf5c700 1 --2- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f6811c5c0 0x7f8f68123880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.724+0000 7f8f6cf5c700 1 --2- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8f68066e70 0x7f8f681352f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.724+0000 7f8f6cf5c700 1 --2- 172.21.15.194:0/3931093037 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8f68067740 0x7f8f681357e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.724+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3931093037 >> 172.21.15.194:0/3931093037 conn(0x7f8f68091bc0 msgr2=0x7f8f68128de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:29.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.724+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3931093037 shutdown_connections 2024-04-26T09:35:29.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.725+0000 7f8f6cf5c700 1 -- 172.21.15.194:0/3931093037 wait complete. 2024-04-26T09:35:29.740 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new be81777d-ab3b-46b3-873e-089f690a2249 1 2024-04-26T09:35:29.846 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:29.840+0000 7fc59c9f1680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T09:35:29.846 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:29.840+0000 7fdf4e007680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T09:35:29.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.894+0000 7f43a3fc9700 1 Processor -- start 2024-04-26T09:35:29.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.894+0000 7f43a3fc9700 1 -- start start 2024-04-26T09:35:29.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a3fc9700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c0670d0 0x7f439c090d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a3fc9700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f439c127330 0x7f439c129720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a3fc9700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f439c067720 con 0x7f439c05e230 2024-04-26T09:35:29.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a3fc9700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f439c091260 con 0x7f439c127330 2024-04-26T09:35:29.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a3fc9700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f439c0913d0 con 0x7f439c0670d0 2024-04-26T09:35:29.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a1564700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c0670d0 0x7f439c090d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a1564700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c0670d0 0x7f439c090d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46910/0 (socket says 172.21.15.194:46910) 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a2566700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f439c127330 0x7f439c129720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a1564700 1 -- 172.21.15.194:0/1241311092 learned_addr learned my addr 172.21.15.194:0/1241311092 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a2566700 1 -- 172.21.15.194:0/1241311092 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c0670d0 msgr2=0x7f439c090d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a2566700 1 --2- 172.21.15.194:0/1241311092 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c0670d0 0x7f439c090d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a2566700 1 -- 172.21.15.194:0/1241311092 >> v1:172.21.15.194:6789/0 conn(0x7f439c05e230 legacy=0x7f439c090660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a2566700 1 -- 172.21.15.194:0/1241311092 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c066b40 con 0x7f439c127330 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a0d63700 1 -- 172.21.15.194:0/1241311092 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864698004 0 0) 0x7f439c067720 con 0x7f439c05e230 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a1564700 1 --2- 172.21.15.194:0/1241311092 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c0670d0 0x7f439c090d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a2566700 1 --2- 172.21.15.194:0/1241311092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f439c127330 0x7f439c129720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4394009e60 tx=0x7f4394009780).ready entity=mon.0 client_cookie=8371bcadb26fa506 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a0d63700 1 -- 172.21.15.194:0/1241311092 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439400e070 con 0x7f439c127330 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a0d63700 1 -- 172.21.15.194:0/1241311092 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439400bb60 con 0x7f439c127330 2024-04-26T09:35:29.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.895+0000 7f43a0d63700 1 -- 172.21.15.194:0/1241311092 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4394012410 con 0x7f439c127330 2024-04-26T09:35:29.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.896+0000 7f43a3fc9700 1 -- 172.21.15.194:0/1241311092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f439c127330 msgr2=0x7f439c129720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.896+0000 7f43a3fc9700 1 --2- 172.21.15.194:0/1241311092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f439c127330 0x7f439c129720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4394009e60 tx=0x7f4394009780).stop 2024-04-26T09:35:29.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.896+0000 7f43a3fc9700 1 -- 172.21.15.194:0/1241311092 shutdown_connections 2024-04-26T09:35:29.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.896+0000 7f43a3fc9700 1 --2- 172.21.15.194:0/1241311092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f439c127330 0x7f439c129720 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.896+0000 7f43a3fc9700 1 --2- 172.21.15.194:0/1241311092 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c0670d0 0x7f439c090d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.896+0000 7f43a3fc9700 1 -- 172.21.15.194:0/1241311092 >> 172.21.15.194:0/1241311092 conn(0x7f439c092180 msgr2=0x7f439c094a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:29.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.896+0000 7f43a3fc9700 1 -- 172.21.15.194:0/1241311092 shutdown_connections 2024-04-26T09:35:29.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.896+0000 7f43a3fc9700 1 -- 172.21.15.194:0/1241311092 wait complete. 2024-04-26T09:35:29.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.897+0000 7f43a3fc9700 1 Processor -- start 2024-04-26T09:35:29.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a3fc9700 1 -- start start 2024-04-26T09:35:29.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a3fc9700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f439c05e230 0x7f439c134d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a3fc9700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c127330 0x7f439c135220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a3fc9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f439c135710 0x7f439c11dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:29.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a3fc9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f439c126640 con 0x7f439c135710 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a3fc9700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f439c126030 con 0x7f439c05e230 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a3fc9700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f439c124bb0 con 0x7f439c127330 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a1d65700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f439c05e230 0x7f439c134d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a2566700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f439c135710 0x7f439c11dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a2566700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f439c135710 0x7f439c11dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:39054/0 (socket says 172.21.15.194:39054) 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a1564700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c127330 0x7f439c135220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a1d65700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f439c05e230 0x7f439c134d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:36706/0 (socket says 172.21.15.194:36706) 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.898+0000 7f43a2566700 1 -- 172.21.15.194:0/651267341 learned_addr learned my addr 172.21.15.194:0/651267341 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43a1564700 1 -- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f439c05e230 msgr2=0x7f439c134d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43a1564700 1 --2- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f439c05e230 0x7f439c134d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43a1564700 1 -- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f439c135710 msgr2=0x7f439c11dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43a1564700 1 --2- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f439c135710 0x7f439c11dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43a1564700 1 -- 172.21.15.194:0/651267341 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c009370 con 0x7f439c127330 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43a1d65700 1 --2- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f439c05e230 0x7f439c134d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:29.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43a2566700 1 --2- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f439c135710 0x7f439c11dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:29.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43a1564700 1 --2- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c127330 0x7f439c135220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f438c006fd0 tx=0x7f438c0090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:29.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43927fc700 1 -- 172.21.15.194:0/651267341 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c00c6a0 con 0x7f439c127330 2024-04-26T09:35:29.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43927fc700 1 -- 172.21.15.194:0/651267341 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f438c00c800 con 0x7f439c127330 2024-04-26T09:35:29.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43a3fc9700 1 -- 172.21.15.194:0/651267341 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4394009a20 con 0x7f439c127330 2024-04-26T09:35:29.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43927fc700 1 -- 172.21.15.194:0/651267341 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f438c00cb80 con 0x7f439c127330 2024-04-26T09:35:29.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43a3fc9700 1 -- 172.21.15.194:0/651267341 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f439c135b70 con 0x7f439c127330 2024-04-26T09:35:29.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43927fc700 1 -- 172.21.15.194:0/651267341 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f438c01b660 con 0x7f439c127330 2024-04-26T09:35:29.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.899+0000 7f43927fc700 1 -- 172.21.15.194:0/651267341 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f438c0127b0 con 0x7f439c127330 2024-04-26T09:35:29.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.900+0000 7f43a3fc9700 1 -- 172.21.15.194:0/651267341 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f439c1269a0 con 0x7f439c127330 2024-04-26T09:35:29.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.903+0000 7f43927fc700 1 -- 172.21.15.194:0/651267341 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f438c03ae50 con 0x7f439c127330 2024-04-26T09:35:29.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.983+0000 7f43a3fc9700 1 -- 172.21.15.194:0/651267341 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "osd new", "uuid": "be81777d-ab3b-46b3-873e-089f690a2249", "id": 1} v 0) v1 -- 0x7f439c095150 con 0x7f439c127330 2024-04-26T09:35:29.994 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.991+0000 7f43927fc700 1 -- 172.21.15.194:0/651267341 <== mon.2 v2:172.21.15.194:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be81777d-ab3b-46b3-873e-089f690a2249", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f438c029030 con 0x7f439c127330 2024-04-26T09:35:29.994 INFO:teuthology.orchestra.run.smithi194.stdout:1 2024-04-26T09:35:29.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.992+0000 7f43a3fc9700 1 -- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c127330 msgr2=0x7f439c135220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:29.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.992+0000 7f43a3fc9700 1 --2- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c127330 0x7f439c135220 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f438c006fd0 tx=0x7f438c0090f0).stop 2024-04-26T09:35:29.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.992+0000 7f43a3fc9700 1 -- 172.21.15.194:0/651267341 shutdown_connections 2024-04-26T09:35:29.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.992+0000 7f43a3fc9700 1 --2- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f439c135710 0x7f439c11dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.992+0000 7f43a3fc9700 1 --2- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f439c05e230 0x7f439c134d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.992+0000 7f43a3fc9700 1 --2- 172.21.15.194:0/651267341 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f439c127330 0x7f439c135220 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:29.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.992+0000 7f43a3fc9700 1 -- 172.21.15.194:0/651267341 >> 172.21.15.194:0/651267341 conn(0x7f439c092180 msgr2=0x7f439c127b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:29.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.992+0000 7f43a3fc9700 1 -- 172.21.15.194:0/651267341 shutdown_connections 2024-04-26T09:35:29.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:29.992+0000 7f43a3fc9700 1 -- 172.21.15.194:0/651267341 wait complete. 2024-04-26T09:35:30.007 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new efddf30f-dc2a-46d5-81d7-0a5cf1b01735 2 2024-04-26T09:35:30.122 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.119+0000 7f4f15b02700 1 Processor -- start 2024-04-26T09:35:30.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.119+0000 7f4f15b02700 1 -- start start 2024-04-26T09:35:30.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.119+0000 7f4f15b02700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101238b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.119+0000 7f4f15b02700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f4f10123da0 0x7f4f1011fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.119+0000 7f4f15b02700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f101232b0 con 0x7f4f101201a0 2024-04-26T09:35:30.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.119+0000 7f4f15b02700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f100688e0 con 0x7f4f10123da0 2024-04-26T09:35:30.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.119+0000 7f4f15b02700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1005e3f0 con 0x7f4f101234d0 2024-04-26T09:35:30.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0f7fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101238b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0effd700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f4f10123da0 0x7f4f1011fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0f7fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101238b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:36714/0 (socket says 172.21.15.194:36714) 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0f7fe700 1 -- 172.21.15.194:0/4073595137 learned_addr learned my addr 172.21.15.194:0/4073595137 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0f7fe700 1 -- 172.21.15.194:0/4073595137 >> v2:172.21.15.194:3300/0 conn(0x7f4f10123da0 msgr2=0x7f4f1011fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0f7fe700 1 --2- 172.21.15.194:0/4073595137 >> v2:172.21.15.194:3300/0 conn(0x7f4f10123da0 0x7f4f1011fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0f7fe700 1 -- 172.21.15.194:0/4073595137 >> v1:172.21.15.194:6789/0 conn(0x7f4f101201a0 legacy=0x7f4f1012a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0f7fe700 1 -- 172.21.15.194:0/4073595137 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f10065b10 con 0x7f4f101234d0 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0e7fc700 1 -- 172.21.15.194:0/4073595137 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951558859 0 0) 0x7f4f101232b0 con 0x7f4f101201a0 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0effd700 1 --2- 172.21.15.194:0/4073595137 >> v2:172.21.15.194:3300/0 conn(0x7f4f10123da0 0x7f4f1011fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0f7fe700 1 --2- 172.21.15.194:0/4073595137 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101238b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4f00009810 tx=0x7f4f00009060).ready entity=mon.1 client_cookie=c0a7c54c0a784213 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0e7fc700 1 -- 172.21.15.194:0/4073595137 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00005880 con 0x7f4f101234d0 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0e7fc700 1 -- 172.21.15.194:0/4073595137 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f000059e0 con 0x7f4f101234d0 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.120+0000 7f4f0e7fc700 1 -- 172.21.15.194:0/4073595137 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00005d60 con 0x7f4f101234d0 2024-04-26T09:35:30.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.121+0000 7f4f15b02700 1 -- 172.21.15.194:0/4073595137 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 msgr2=0x7f4f101238b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.121+0000 7f4f15b02700 1 --2- 172.21.15.194:0/4073595137 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101238b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4f00009810 tx=0x7f4f00009060).stop 2024-04-26T09:35:30.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.121+0000 7f4f15b02700 1 -- 172.21.15.194:0/4073595137 shutdown_connections 2024-04-26T09:35:30.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.121+0000 7f4f15b02700 1 --2- 172.21.15.194:0/4073595137 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101238b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.121+0000 7f4f15b02700 1 --2- 172.21.15.194:0/4073595137 >> v2:172.21.15.194:3300/0 conn(0x7f4f10123da0 0x7f4f1011fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.121+0000 7f4f15b02700 1 -- 172.21.15.194:0/4073595137 >> 172.21.15.194:0/4073595137 conn(0x7f4f10092180 msgr2=0x7f4f10094a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:30.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.121+0000 7f4f15b02700 1 -- 172.21.15.194:0/4073595137 shutdown_connections 2024-04-26T09:35:30.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.121+0000 7f4f15b02700 1 -- 172.21.15.194:0/4073595137 wait complete. 2024-04-26T09:35:30.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.122+0000 7f4f15b02700 1 Processor -- start 2024-04-26T09:35:30.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.122+0000 7f4f15b02700 1 -- start start 2024-04-26T09:35:30.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.122+0000 7f4f15b02700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4f101201a0 0x7f4f101354e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.122+0000 7f4f15b02700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101359d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.122+0000 7f4f15b02700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f10123da0 0x7f4f100665b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.122+0000 7f4f15b02700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f10089670 con 0x7f4f10123da0 2024-04-26T09:35:30.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.122+0000 7f4f15b02700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f100892f0 con 0x7f4f101234d0 2024-04-26T09:35:30.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.122+0000 7f4f15b02700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4f10088f20 con 0x7f4f101201a0 2024-04-26T09:35:30.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0effd700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0effd700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101359d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:36720/0 (socket says 172.21.15.194:36720) 2024-04-26T09:35:30.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0effd700 1 -- 172.21.15.194:0/4118318856 learned_addr learned my addr 172.21.15.194:0/4118318856 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:30.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0ffff700 1 --2- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f10123da0 0x7f4f100665b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0f7fe700 1 --2- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4f101201a0 0x7f4f101354e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0effd700 1 -- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4f101201a0 msgr2=0x7f4f101354e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0effd700 1 --2- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4f101201a0 0x7f4f101354e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0effd700 1 -- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f10123da0 msgr2=0x7f4f100665b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0effd700 1 --2- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f10123da0 0x7f4f100665b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0effd700 1 -- 172.21.15.194:0/4118318856 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f040092a0 con 0x7f4f101234d0 2024-04-26T09:35:30.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0ffff700 1 --2- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f10123da0 0x7f4f100665b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:30.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f0effd700 1 --2- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101359d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4f0400e9d0 tx=0x7f4f04009060).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:30.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4ef7fff700 1 -- 172.21.15.194:0/4118318856 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04009600 con 0x7f4f101234d0 2024-04-26T09:35:30.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4ef7fff700 1 -- 172.21.15.194:0/4118318856 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0400c6a0 con 0x7f4f101234d0 2024-04-26T09:35:30.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4ef7fff700 1 -- 172.21.15.194:0/4118318856 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0400ca20 con 0x7f4f101234d0 2024-04-26T09:35:30.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f15b02700 1 -- 172.21.15.194:0/4118318856 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f00009300 con 0x7f4f101234d0 2024-04-26T09:35:30.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4f15b02700 1 -- 172.21.15.194:0/4118318856 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f10135ec0 con 0x7f4f101234d0 2024-04-26T09:35:30.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.123+0000 7f4ef7fff700 1 -- 172.21.15.194:0/4118318856 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f4f0401b400 con 0x7f4f101234d0 2024-04-26T09:35:30.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.124+0000 7f4ef7fff700 1 -- 172.21.15.194:0/4118318856 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f4f04012820 con 0x7f4f101234d0 2024-04-26T09:35:30.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.124+0000 7f4f15b02700 1 -- 172.21.15.194:0/4118318856 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f100681b0 con 0x7f4f101234d0 2024-04-26T09:35:30.132 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.127+0000 7f4ef7fff700 1 -- 172.21.15.194:0/4118318856 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f4f0403ae50 con 0x7f4f101234d0 2024-04-26T09:35:30.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.217+0000 7f4f15b02700 1 -- 172.21.15.194:0/4118318856 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "osd new", "uuid": "efddf30f-dc2a-46d5-81d7-0a5cf1b01735", "id": 2} v 0) v1 -- 0x7f4f10136140 con 0x7f4f101234d0 2024-04-26T09:35:30.224 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.221+0000 7f4ef7fff700 1 -- 172.21.15.194:0/4118318856 <== mon.1 v2:172.21.15.194:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "efddf30f-dc2a-46d5-81d7-0a5cf1b01735", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4f04029030 con 0x7f4f101234d0 2024-04-26T09:35:30.224 INFO:teuthology.orchestra.run.smithi194.stdout:2 2024-04-26T09:35:30.225 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.221+0000 7f4f15b02700 1 -- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 msgr2=0x7f4f101359d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.225 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.221+0000 7f4f15b02700 1 --2- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101359d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4f0400e9d0 tx=0x7f4f04009060).stop 2024-04-26T09:35:30.225 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.222+0000 7f4f15b02700 1 -- 172.21.15.194:0/4118318856 shutdown_connections 2024-04-26T09:35:30.225 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.222+0000 7f4f15b02700 1 --2- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f10123da0 0x7f4f100665b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.225 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.222+0000 7f4f15b02700 1 --2- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4f101201a0 0x7f4f101354e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.225 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.222+0000 7f4f15b02700 1 --2- 172.21.15.194:0/4118318856 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f101234d0 0x7f4f101359d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.222+0000 7f4f15b02700 1 -- 172.21.15.194:0/4118318856 >> 172.21.15.194:0/4118318856 conn(0x7f4f10092180 msgr2=0x7f4f10119b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:30.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.226+0000 7f4f15b02700 1 -- 172.21.15.194:0/4118318856 shutdown_connections 2024-04-26T09:35:30.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.226+0000 7f4f15b02700 1 -- 172.21.15.194:0/4118318856 wait complete. 2024-04-26T09:35:30.241 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 488b4848-1a54-4d45-b95d-ff5ccc61af12 3 2024-04-26T09:35:30.289 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:30.285+0000 7fc59c9f1680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T09:35:30.289 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:30.286+0000 7fdf4e007680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T09:35:30.358 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f45a5d16700 1 Processor -- start 2024-04-26T09:35:30.358 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f45a5d16700 1 -- start start 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f45a5d16700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f45a0090b40 0x7f45a0090f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f45a5d16700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f45a012ac60 0x7f45a012d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f45a5d16700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a008ef10 con 0x7f45a005e230 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f45a5d16700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45a0069e10 con 0x7f45a0090b40 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f45a5d16700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a00654a0 con 0x7f45a012ac60 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f459effd700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f45a0090b40 0x7f45a0090f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f459effd700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f45a0090b40 0x7f45a0090f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:39088/0 (socket says 172.21.15.194:39088) 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f459effd700 1 -- 172.21.15.194:0/715649505 learned_addr learned my addr 172.21.15.194:0/715649505 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f459f7fe700 1 --1- 172.21.15.194:0/715649505 >> v1:172.21.15.194:6789/0 conn(0x7f45a005e230 0x7f45a0090430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:57044/0 (socket says 172.21.15.194:57044) 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.353+0000 7f459ffff700 1 --2- 172.21.15.194:0/715649505 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f45a012ac60 0x7f45a012d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.354+0000 7f459effd700 1 -- 172.21.15.194:0/715649505 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f45a012ac60 msgr2=0x7f45a012d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.354+0000 7f459effd700 1 --2- 172.21.15.194:0/715649505 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f45a012ac60 0x7f45a012d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.354+0000 7f459effd700 1 -- 172.21.15.194:0/715649505 >> v1:172.21.15.194:6789/0 conn(0x7f45a005e230 legacy=0x7f45a0090430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.354+0000 7f459effd700 1 -- 172.21.15.194:0/715649505 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a00667b0 con 0x7f45a0090b40 2024-04-26T09:35:30.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.354+0000 7f459e7fc700 1 -- 172.21.15.194:0/715649505 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770319309 0 0) 0x7f45a008ef10 con 0x7f45a005e230 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.354+0000 7f459effd700 1 --2- 172.21.15.194:0/715649505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45a0090b40 0x7f45a0090f20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4594006fd0 tx=0x7f45940090f0).ready entity=mon.0 client_cookie=93b2aad09f1074d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.354+0000 7f459e7fc700 1 -- 172.21.15.194:0/715649505 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459400b580 con 0x7f45a0090b40 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.354+0000 7f459e7fc700 1 -- 172.21.15.194:0/715649505 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459400b6e0 con 0x7f45a0090b40 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.354+0000 7f459e7fc700 1 -- 172.21.15.194:0/715649505 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459400ba60 con 0x7f45a0090b40 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.355+0000 7f45a5d16700 1 -- 172.21.15.194:0/715649505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45a0090b40 msgr2=0x7f45a0090f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.355+0000 7f45a5d16700 1 --2- 172.21.15.194:0/715649505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45a0090b40 0x7f45a0090f20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f4594006fd0 tx=0x7f45940090f0).stop 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.355+0000 7f45a5d16700 1 -- 172.21.15.194:0/715649505 shutdown_connections 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.355+0000 7f45a5d16700 1 --2- 172.21.15.194:0/715649505 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f45a012ac60 0x7f45a012d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.355+0000 7f45a5d16700 1 --2- 172.21.15.194:0/715649505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45a0090b40 0x7f45a0090f20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.355+0000 7f45a5d16700 1 -- 172.21.15.194:0/715649505 >> 172.21.15.194:0/715649505 conn(0x7f45a0092180 msgr2=0x7f45a0094a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.355+0000 7f45a5d16700 1 -- 172.21.15.194:0/715649505 shutdown_connections 2024-04-26T09:35:30.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.355+0000 7f45a5d16700 1 -- 172.21.15.194:0/715649505 wait complete. 2024-04-26T09:35:30.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.356+0000 7f45a5d16700 1 Processor -- start 2024-04-26T09:35:30.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.356+0000 7f45a5d16700 1 -- start start 2024-04-26T09:35:30.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f45a5d16700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f45a005e230 0x7f45a011a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f45a5d16700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45a0090b40 0x7f45a011a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f45a5d16700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f45a012ac60 0x7f45a011ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f45a5d16700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a012a7f0 con 0x7f45a0090b40 2024-04-26T09:35:30.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f45a5d16700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a011d6b0 con 0x7f45a012ac60 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f45a5d16700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f45a011da80 con 0x7f45a005e230 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45a0090b40 0x7f45a011a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459effd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45a0090b40 0x7f45a011a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:39098/0 (socket says 172.21.15.194:39098) 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459effd700 1 -- 172.21.15.194:0/1057812867 learned_addr learned my addr 172.21.15.194:0/1057812867 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459f7fe700 1 --2- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f45a005e230 0x7f45a011a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459ffff700 1 --2- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f45a012ac60 0x7f45a011ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459effd700 1 -- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f45a005e230 msgr2=0x7f45a011a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459effd700 1 --2- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f45a005e230 0x7f45a011a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459effd700 1 -- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f45a012ac60 msgr2=0x7f45a011ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459effd700 1 --2- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f45a012ac60 0x7f45a011ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459effd700 1 -- 172.21.15.194:0/1057812867 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4594009370 con 0x7f45a0090b40 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459ffff700 1 --2- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f45a012ac60 0x7f45a011ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f459effd700 1 --2- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45a0090b40 0x7f45a011a680 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4594009130 tx=0x7f4594017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f4587fff700 1 -- 172.21.15.194:0/1057812867 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459400b850 con 0x7f45a0090b40 2024-04-26T09:35:30.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f4587fff700 1 -- 172.21.15.194:0/1057812867 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459400b9b0 con 0x7f45a0090b40 2024-04-26T09:35:30.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f4587fff700 1 -- 172.21.15.194:0/1057812867 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f459401c670 con 0x7f45a0090b40 2024-04-26T09:35:30.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f45a5d16700 1 -- 172.21.15.194:0/1057812867 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a011cff0 con 0x7f45a0090b40 2024-04-26T09:35:30.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.357+0000 7f45a5d16700 1 -- 172.21.15.194:0/1057812867 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a011b060 con 0x7f45a0090b40 2024-04-26T09:35:30.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.358+0000 7f4587fff700 1 -- 172.21.15.194:0/1057812867 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f459401cc40 con 0x7f45a0090b40 2024-04-26T09:35:30.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.358+0000 7f4587fff700 1 -- 172.21.15.194:0/1057812867 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f459401b850 con 0x7f45a0090b40 2024-04-26T09:35:30.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.359+0000 7f45a5d16700 1 -- 172.21.15.194:0/1057812867 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a011d210 con 0x7f45a0090b40 2024-04-26T09:35:30.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.361+0000 7f4587fff700 1 -- 172.21.15.194:0/1057812867 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f4594044e50 con 0x7f45a0090b40 2024-04-26T09:35:30.396 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:30.392+0000 7fdf4e007680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T09:35:30.396 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:30.393+0000 7fc59c9f1680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T09:35:30.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.440+0000 7f45a5d16700 1 -- 172.21.15.194:0/1057812867 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd new", "uuid": "488b4848-1a54-4d45-b95d-ff5ccc61af12", "id": 3} v 0) v1 -- 0x7f45a00659f0 con 0x7f45a0090b40 2024-04-26T09:35:30.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.443+0000 7f4587fff700 1 -- 172.21.15.194:0/1057812867 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "488b4848-1a54-4d45-b95d-ff5ccc61af12", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4594033070 con 0x7f45a0090b40 2024-04-26T09:35:30.446 INFO:teuthology.orchestra.run.smithi194.stdout:3 2024-04-26T09:35:30.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.444+0000 7f45a5d16700 1 -- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45a0090b40 msgr2=0x7f45a011a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.444+0000 7f45a5d16700 1 --2- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45a0090b40 0x7f45a011a680 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4594009130 tx=0x7f4594017000).stop 2024-04-26T09:35:30.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.445+0000 7f45a5d16700 1 -- 172.21.15.194:0/1057812867 shutdown_connections 2024-04-26T09:35:30.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.445+0000 7f45a5d16700 1 --2- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f45a012ac60 0x7f45a011ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.445+0000 7f45a5d16700 1 --2- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f45a005e230 0x7f45a011a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.445+0000 7f45a5d16700 1 --2- 172.21.15.194:0/1057812867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45a0090b40 0x7f45a011a680 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.445+0000 7f45a5d16700 1 -- 172.21.15.194:0/1057812867 >> 172.21.15.194:0/1057812867 conn(0x7f45a0092180 msgr2=0x7f45a0092850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:30.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.445+0000 7f45a5d16700 1 -- 172.21.15.194:0/1057812867 shutdown_connections 2024-04-26T09:35:30.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.445+0000 7f45a5d16700 1 -- 172.21.15.194:0/1057812867 wait complete. 2024-04-26T09:35:30.460 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-26T09:35:30.460 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-26T09:35:30.462 INFO:tasks.ceph.osd.0:Started 2024-04-26T09:35:30.463 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-26T09:35:30.463 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-26T09:35:30.463 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:30.460+0000 7fdf4e007680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T09:35:30.464 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:30.460+0000 7fc59c9f1680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T09:35:30.465 INFO:tasks.ceph.osd.1:Started 2024-04-26T09:35:30.466 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-26T09:35:30.466 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-26T09:35:30.468 INFO:tasks.ceph.osd.2:Started 2024-04-26T09:35:30.468 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-26T09:35:30.468 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-26T09:35:30.470 INFO:tasks.ceph.osd.3:Started 2024-04-26T09:35:30.471 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T09:35:30.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.629+0000 7f8c083ed700 1 Processor -- start 2024-04-26T09:35:30.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.629+0000 7f8c083ed700 1 -- start start 2024-04-26T09:35:30.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.629+0000 7f8c083ed700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 0x7f8c00125a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.629+0000 7f8c083ed700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00125f20 0x7f8c00064a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c083ed700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c00069830 con 0x7f8c00064f00 2024-04-26T09:35:30.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c083ed700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c00065df0 con 0x7f8c00125650 2024-04-26T09:35:30.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c083ed700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8c000674c0 con 0x7f8c00125f20 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05988700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00125f20 0x7f8c00064a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05988700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00125f20 0x7f8c00064a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46990/0 (socket says 172.21.15.194:46990) 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05988700 1 -- 172.21.15.194:0/440805172 learned_addr learned my addr 172.21.15.194:0/440805172 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c06189700 1 --2- 172.21.15.194:0/440805172 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 0x7f8c00125a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05988700 1 -- 172.21.15.194:0/440805172 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 msgr2=0x7f8c00125a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05187700 1 -- 172.21.15.194:0/440805172 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505292367 0 0) 0x7f8c00069830 con 0x7f8c00064f00 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05988700 1 --2- 172.21.15.194:0/440805172 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 0x7f8c00125a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05988700 1 -- 172.21.15.194:0/440805172 >> v1:172.21.15.194:6789/0 conn(0x7f8c00064f00 legacy=0x7f8c0012c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05988700 1 -- 172.21.15.194:0/440805172 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c001252b0 con 0x7f8c00125f20 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c06189700 1 --2- 172.21.15.194:0/440805172 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 0x7f8c00125a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05988700 1 --2- 172.21.15.194:0/440805172 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00125f20 0x7f8c00064a10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8bf0009830 tx=0x7f8bf00090f0).ready entity=mon.2 client_cookie=9837dbbb15989cbc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05187700 1 -- 172.21.15.194:0/440805172 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf000b520 con 0x7f8c00125f20 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05187700 1 -- 172.21.15.194:0/440805172 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf000b680 con 0x7f8c00125f20 2024-04-26T09:35:30.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.630+0000 7f8c05187700 1 -- 172.21.15.194:0/440805172 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf000ba20 con 0x7f8c00125f20 2024-04-26T09:35:30.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.631+0000 7f8c083ed700 1 -- 172.21.15.194:0/440805172 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00125f20 msgr2=0x7f8c00064a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.631+0000 7f8c083ed700 1 --2- 172.21.15.194:0/440805172 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00125f20 0x7f8c00064a10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8bf0009830 tx=0x7f8bf00090f0).stop 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.631+0000 7f8c083ed700 1 -- 172.21.15.194:0/440805172 shutdown_connections 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.631+0000 7f8c083ed700 1 --2- 172.21.15.194:0/440805172 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 0x7f8c00125a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.631+0000 7f8c083ed700 1 --2- 172.21.15.194:0/440805172 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00125f20 0x7f8c00064a10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.631+0000 7f8c083ed700 1 -- 172.21.15.194:0/440805172 >> 172.21.15.194:0/440805172 conn(0x7f8c00096a50 msgr2=0x7f8c00091a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.631+0000 7f8c083ed700 1 -- 172.21.15.194:0/440805172 shutdown_connections 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.631+0000 7f8c083ed700 1 -- 172.21.15.194:0/440805172 wait complete. 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c083ed700 1 Processor -- start 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c083ed700 1 -- start start 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c083ed700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00064f00 0x7f8c001186b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c083ed700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 0x7f8c00118ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c083ed700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c00125f20 0x7f8c00119090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c083ed700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0011bbd0 con 0x7f8c00125f20 2024-04-26T09:35:30.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c083ed700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0011bfa0 con 0x7f8c00125650 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c083ed700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0011a2c0 con 0x7f8c00064f00 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c06189700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00064f00 0x7f8c001186b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c05988700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 0x7f8c00118ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c06189700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00064f00 0x7f8c001186b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46998/0 (socket says 172.21.15.194:46998) 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c06189700 1 -- 172.21.15.194:0/2654870531 learned_addr learned my addr 172.21.15.194:0/2654870531 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.632+0000 7f8c0698a700 1 --2- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c00125f20 0x7f8c00119090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8c05988700 1 -- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00064f00 msgr2=0x7f8c001186b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8c05988700 1 --2- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00064f00 0x7f8c001186b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8c05988700 1 -- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c00125f20 msgr2=0x7f8c00119090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8c05988700 1 --2- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c00125f20 0x7f8c00119090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8c05988700 1 -- 172.21.15.194:0/2654870531 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc0092a0 con 0x7f8c00125650 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8c06189700 1 --2- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00064f00 0x7f8c001186b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8c0698a700 1 --2- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c00125f20 0x7f8c00119090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:30.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8c05988700 1 --2- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 0x7f8c00118ba0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8bf0014000 tx=0x7f8bf0017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:30.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8bf6ffd700 1 -- 172.21.15.194:0/2654870531 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0006d90 con 0x7f8c00125650 2024-04-26T09:35:30.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8bf6ffd700 1 -- 172.21.15.194:0/2654870531 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf0006910 con 0x7f8c00125650 2024-04-26T09:35:30.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8c083ed700 1 -- 172.21.15.194:0/2654870531 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf0009370 con 0x7f8c00125650 2024-04-26T09:35:30.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8bf6ffd700 1 -- 172.21.15.194:0/2654870531 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf000b7f0 con 0x7f8c00125650 2024-04-26T09:35:30.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.633+0000 7f8c083ed700 1 -- 172.21.15.194:0/2654870531 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0011b5b0 con 0x7f8c00125650 2024-04-26T09:35:30.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.634+0000 7f8bf6ffd700 1 -- 172.21.15.194:0/2654870531 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f8bf001c890 con 0x7f8c00125650 2024-04-26T09:35:30.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.634+0000 7f8bf6ffd700 1 -- 172.21.15.194:0/2654870531 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f8bf000b9a0 con 0x7f8c00125650 2024-04-26T09:35:30.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.634+0000 7f8c083ed700 1 -- 172.21.15.194:0/2654870531 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0011b740 con 0x7f8c00125650 2024-04-26T09:35:30.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.636+0000 7f8bf6ffd700 1 -- 172.21.15.194:0/2654870531 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f8bf0044e50 con 0x7f8c00125650 2024-04-26T09:35:30.652 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:30.649+0000 7fc59c9f1680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T09:35:30.653 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:30.649+0000 7fdf4e007680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T09:35:30.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.704+0000 7f8c083ed700 1 -- 172.21.15.194:0/2654870531 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c001250d0 con 0x7f8c00125650 2024-04-26T09:35:30.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.704+0000 7f8bf6ffd700 1 -- 172.21.15.194:0/2654870531 <== mon.1 v2:172.21.15.194:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4109 (secure 0 0 0) 0x7f8bf0033030 con 0x7f8c00125650 2024-04-26T09:35:30.708 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:30.708 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":6,"fsid":"be7f6752-916a-4f92-bdb5-9db81c891e5c","created":"2024-04-26T09:35:29.120276+0000","modified":"2024-04-26T09:35:30.442422+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T09:35:30.442422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"79a6c3fb-32e2-46b3-81f7-7ab1e3bf3e5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"be81777d-ab3b-46b3-873e-089f690a2249","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efddf30f-dc2a-46d5-81d7-0a5cf1b01735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"488b4848-1a54-4d45-b95d-ff5ccc61af12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T09:35:30.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.705+0000 7f8c083ed700 1 -- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 msgr2=0x7f8c00118ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:30.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.705+0000 7f8c083ed700 1 --2- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 0x7f8c00118ba0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f8bf0014000 tx=0x7f8bf0017000).stop 2024-04-26T09:35:30.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.706+0000 7f8c083ed700 1 -- 172.21.15.194:0/2654870531 shutdown_connections 2024-04-26T09:35:30.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.706+0000 7f8c083ed700 1 --2- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8c00125f20 0x7f8c00119090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.706+0000 7f8c083ed700 1 --2- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f8c00064f00 0x7f8c001186b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.706+0000 7f8c083ed700 1 --2- 172.21.15.194:0/2654870531 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f8c00125650 0x7f8c00118ba0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:30.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.706+0000 7f8c083ed700 1 -- 172.21.15.194:0/2654870531 >> 172.21.15.194:0/2654870531 conn(0x7f8c00096a50 msgr2=0x7f8c00068d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:30.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.706+0000 7f8c083ed700 1 -- 172.21.15.194:0/2654870531 shutdown_connections 2024-04-26T09:35:30.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:30.706+0000 7f8c083ed700 1 -- 172.21.15.194:0/2654870531 wait complete. 2024-04-26T09:35:30.721 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-26T09:35:30.721 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-26T09:35:30.728 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:30.724+0000 7fc59c9f1680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T09:35:30.728 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:30.725+0000 7fdf4e007680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T09:35:31.090 INFO:tasks.ceph.osd.1.smithi194.stderr:2024-04-26T09:35:31.086+0000 7f466215e380 -1 Falling back to public interface 2024-04-26T09:35:31.108 INFO:tasks.ceph.osd.3.smithi194.stderr:2024-04-26T09:35:31.105+0000 7f659880e380 -1 Falling back to public interface 2024-04-26T09:35:31.110 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:31.107+0000 7fdf4e007680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T09:35:31.111 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:31.107+0000 7fc59c9f1680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T09:35:31.112 INFO:tasks.ceph.osd.0.smithi194.stderr:2024-04-26T09:35:31.109+0000 7f73aeeb4380 -1 Falling back to public interface 2024-04-26T09:35:31.123 DEBUG:teuthology.orchestra.run.smithi194:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T09:35:31.128 INFO:tasks.ceph.osd.2.smithi194.stderr:2024-04-26T09:35:31.125+0000 7fee797ae380 -1 Falling back to public interface 2024-04-26T09:35:31.207 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:31.204+0000 7fdf4e007680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T09:35:31.212 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:31.209+0000 7fc59c9f1680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T09:35:31.233 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.230+0000 7f26edfaa700 1 Processor -- start 2024-04-26T09:35:31.233 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.230+0000 7f26edfaa700 1 -- start start 2024-04-26T09:35:31.236 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.230+0000 7f26edfaa700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e81223e0 0x7f26e81227c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:31.236 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.230+0000 7f26edfaa700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f26e8122cb0 0x7f26e8067630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:31.236 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.230+0000 7f26edfaa700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f26e8067b20 0x7f26e8067fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:31.236 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.230+0000 7f26edfaa700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e8122020 con 0x7f26e8067b20 2024-04-26T09:35:31.236 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.230+0000 7f26edfaa700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8069060 con 0x7f26e8122cb0 2024-04-26T09:35:31.236 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.230+0000 7f26edfaa700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80684a0 con 0x7f26e81223e0 2024-04-26T09:35:31.236 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.230+0000 7f26e77fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e81223e0 0x7f26e81227c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:31.236 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e77fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e81223e0 0x7f26e81227c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47014/0 (socket says 172.21.15.194:47014) 2024-04-26T09:35:31.236 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e77fe700 1 -- 172.21.15.194:0/1317166848 learned_addr learned my addr 172.21.15.194:0/1317166848 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e7fff700 1 --2- 172.21.15.194:0/1317166848 >> v2:172.21.15.194:3300/0 conn(0x7f26e8067b20 0x7f26e8067fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e6ffd700 1 --2- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f26e8122cb0 0x7f26e8067630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e7fff700 1 -- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e81223e0 msgr2=0x7f26e81227c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e7fff700 1 --2- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e81223e0 0x7f26e81227c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e7fff700 1 -- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f26e8122cb0 msgr2=0x7f26e8067630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e7fff700 1 --2- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f26e8122cb0 0x7f26e8067630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e7fff700 1 -- 172.21.15.194:0/1317166848 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e8069a40 con 0x7f26e8067b20 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e77fe700 1 --2- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e81223e0 0x7f26e81227c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e7fff700 1 --2- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26e8067b20 0x7f26e8067fb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f26e0009e60 tx=0x7f26e0009780).ready entity=mon.0 client_cookie=a8b32aef72c5b1f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e67fc700 1 -- 172.21.15.194:0/1317166848 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e000e070 con 0x7f26e8067b20 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e67fc700 1 -- 172.21.15.194:0/1317166848 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e000ba50 con 0x7f26e8067b20 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.231+0000 7f26e67fc700 1 -- 172.21.15.194:0/1317166848 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e000bdd0 con 0x7f26e8067b20 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.232+0000 7f26edfaa700 1 -- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26e8067b20 msgr2=0x7f26e8067fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.232+0000 7f26edfaa700 1 --2- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26e8067b20 0x7f26e8067fb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f26e0009e60 tx=0x7f26e0009780).stop 2024-04-26T09:35:31.237 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.232+0000 7f26edfaa700 1 -- 172.21.15.194:0/1317166848 shutdown_connections 2024-04-26T09:35:31.238 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.232+0000 7f26edfaa700 1 --2- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26e8067b20 0x7f26e8067fb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:31.238 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.232+0000 7f26edfaa700 1 --2- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e81223e0 0x7f26e81227c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:31.238 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.232+0000 7f26edfaa700 1 --2- 172.21.15.194:0/1317166848 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f26e8122cb0 0x7f26e8067630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:31.238 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.232+0000 7f26edfaa700 1 -- 172.21.15.194:0/1317166848 >> 172.21.15.194:0/1317166848 conn(0x7f26e8094650 msgr2=0x7f26e8094a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:31.238 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.232+0000 7f26edfaa700 1 -- 172.21.15.194:0/1317166848 shutdown_connections 2024-04-26T09:35:31.238 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.232+0000 7f26edfaa700 1 -- 172.21.15.194:0/1317166848 wait complete. 2024-04-26T09:35:31.238 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.233+0000 7f26edfaa700 1 Processor -- start 2024-04-26T09:35:31.240 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.233+0000 7f26edfaa700 1 -- start start 2024-04-26T09:35:31.240 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26edfaa700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e8067b20 0x7f26e8090a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:31.240 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26edfaa700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26e81223e0 0x7f26e80873a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:31.240 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26edfaa700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f26e8122cb0 0x7f26e8087890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:31.240 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26edfaa700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8090ef0 con 0x7f26e81223e0 2024-04-26T09:35:31.240 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26edfaa700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8089b50 con 0x7f26e8122cb0 2024-04-26T09:35:31.240 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26edfaa700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f26e80891f0 con 0x7f26e8067b20 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e77fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e8067b20 0x7f26e8090a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e6ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26e81223e0 0x7f26e80873a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e77fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e8067b20 0x7f26e8090a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47016/0 (socket says 172.21.15.194:47016) 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e77fe700 1 -- 172.21.15.194:0/24917545 learned_addr learned my addr 172.21.15.194:0/24917545 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e7fff700 1 --2- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f26e8122cb0 0x7f26e8087890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e6ffd700 1 -- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e8067b20 msgr2=0x7f26e8090a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e6ffd700 1 --2- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e8067b20 0x7f26e8090a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e6ffd700 1 -- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f26e8122cb0 msgr2=0x7f26e8087890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e6ffd700 1 --2- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f26e8122cb0 0x7f26e8087890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e6ffd700 1 -- 172.21.15.194:0/24917545 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d800b610 con 0x7f26e81223e0 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e77fe700 1 --2- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e8067b20 0x7f26e8090a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26e6ffd700 1 --2- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26e81223e0 0x7f26e80873a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f26e8064e20 tx=0x7f26dc009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:31.241 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26cffff700 1 -- 172.21.15.194:0/24917545 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc009450 con 0x7f26e81223e0 2024-04-26T09:35:31.242 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26cffff700 1 -- 172.21.15.194:0/24917545 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26dc012600 con 0x7f26e81223e0 2024-04-26T09:35:31.242 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26cffff700 1 -- 172.21.15.194:0/24917545 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26dc012980 con 0x7f26e81223e0 2024-04-26T09:35:31.242 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26edfaa700 1 -- 172.21.15.194:0/24917545 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e0009a20 con 0x7f26e81223e0 2024-04-26T09:35:31.242 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.234+0000 7f26edfaa700 1 -- 172.21.15.194:0/24917545 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e811f820 con 0x7f26e81223e0 2024-04-26T09:35:31.242 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.235+0000 7f26cffff700 1 -- 172.21.15.194:0/24917545 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) v1 ==== 495+0+0 (secure 0 0 0) 0x7f26dc01b400 con 0x7f26e81223e0 2024-04-26T09:35:31.242 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.235+0000 7f26cffff700 1 -- 172.21.15.194:0/24917545 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f26dc01a3b0 con 0x7f26e81223e0 2024-04-26T09:35:31.242 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.236+0000 7f26edfaa700 1 -- 172.21.15.194:0/24917545 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e805a310 con 0x7f26e81223e0 2024-04-26T09:35:31.243 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.240+0000 7f26cffff700 1 -- 172.21.15.194:0/24917545 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73271 (secure 0 0 0) 0x7f26dc016020 con 0x7f26e81223e0 2024-04-26T09:35:31.312 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.308+0000 7f26edfaa700 1 -- 172.21.15.194:0/24917545 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26e80881a0 con 0x7f26e81223e0 2024-04-26T09:35:31.312 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.308+0000 7f26cffff700 1 -- 172.21.15.194:0/24917545 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) v1 ==== 74+0+4109 (secure 0 0 0) 0x7f26dc01b770 con 0x7f26e81223e0 2024-04-26T09:35:31.312 INFO:teuthology.misc.health.smithi194.stdout: 2024-04-26T09:35:31.312 INFO:teuthology.misc.health.smithi194.stdout:{"epoch":6,"fsid":"be7f6752-916a-4f92-bdb5-9db81c891e5c","created":"2024-04-26T09:35:29.120276+0000","modified":"2024-04-26T09:35:30.442422+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T09:35:30.442422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"79a6c3fb-32e2-46b3-81f7-7ab1e3bf3e5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"be81777d-ab3b-46b3-873e-089f690a2249","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"efddf30f-dc2a-46d5-81d7-0a5cf1b01735","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"488b4848-1a54-4d45-b95d-ff5ccc61af12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T09:35:31.313 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.309+0000 7f26edfaa700 1 -- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26e81223e0 msgr2=0x7f26e80873a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:31.313 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.309+0000 7f26edfaa700 1 --2- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26e81223e0 0x7f26e80873a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f26e8064e20 tx=0x7f26dc009060).stop 2024-04-26T09:35:31.313 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.309+0000 7f26edfaa700 1 -- 172.21.15.194:0/24917545 shutdown_connections 2024-04-26T09:35:31.313 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.309+0000 7f26edfaa700 1 --2- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f26e8122cb0 0x7f26e8087890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:31.313 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.309+0000 7f26edfaa700 1 --2- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f26e8067b20 0x7f26e8090a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:31.313 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.309+0000 7f26edfaa700 1 --2- 172.21.15.194:0/24917545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f26e81223e0 0x7f26e80873a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:31.314 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.310+0000 7f26edfaa700 1 -- 172.21.15.194:0/24917545 >> 172.21.15.194:0/24917545 conn(0x7f26e8094650 msgr2=0x7f26e8065990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:31.314 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.310+0000 7f26edfaa700 1 -- 172.21.15.194:0/24917545 shutdown_connections 2024-04-26T09:35:31.314 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:31.310+0000 7f26edfaa700 1 -- 172.21.15.194:0/24917545 wait complete. 2024-04-26T09:35:31.324 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-04-26T09:35:31.549 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:31.545+0000 7fc59c9f1680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T09:35:31.549 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:31.545+0000 7fdf4e007680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T09:35:31.628 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:31.624+0000 7fc59c9f1680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T09:35:31.628 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:31.625+0000 7fdf4e007680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T09:35:31.777 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:31.774+0000 7fc59c9f1680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T09:35:31.778 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:31.774+0000 7fdf4e007680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T09:35:31.848 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:31.845+0000 7fc59c9f1680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T09:35:31.849 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:31.845+0000 7fdf4e007680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T09:35:32.095 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:32.091+0000 7fc59c9f1680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T09:35:32.096 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:32.091+0000 7fdf4e007680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T09:35:32.205 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:32.201+0000 7fc59c9f1680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T09:35:32.205 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:32.202+0000 7fdf4e007680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T09:35:32.358 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:32.355+0000 7fdf4e007680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T09:35:32.359 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:32.355+0000 7fc59c9f1680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T09:35:32.430 INFO:tasks.ceph.mgr.y.smithi194.stderr:2024-04-26T09:35:32.426+0000 7fc59c9f1680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T09:35:32.430 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:32.427+0000 7fdf4e007680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T09:35:33.444 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:33.440+0000 7fdf25073700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-26T09:35:33.444 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-04-26T09:35:33.440+0000 7fdf25073700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T09:35:34.585 INFO:tasks.ceph.osd.1.smithi194.stderr:2024-04-26T09:35:34.581+0000 7f466215e380 -1 osd.1 0 log_to_monitors {default=true} 2024-04-26T09:35:34.588 INFO:tasks.ceph.osd.0.smithi194.stderr:2024-04-26T09:35:34.585+0000 7f73aeeb4380 -1 osd.0 0 log_to_monitors {default=true} 2024-04-26T09:35:34.612 INFO:tasks.ceph.osd.3.smithi194.stderr:2024-04-26T09:35:34.608+0000 7f659880e380 -1 osd.3 0 log_to_monitors {default=true} 2024-04-26T09:35:34.634 INFO:tasks.ceph.osd.2.smithi194.stderr:2024-04-26T09:35:34.631+0000 7fee797ae380 -1 osd.2 0 log_to_monitors {default=true} 2024-04-26T09:35:36.454 INFO:tasks.ceph.osd.0.smithi194.stderr:2024-04-26T09:35:36.450+0000 7f73a5855700 -1 osd.0 0 waiting for initial osdmap 2024-04-26T09:35:36.454 INFO:tasks.ceph.osd.3.smithi194.stderr:2024-04-26T09:35:36.450+0000 7f658f1af700 -1 osd.3 0 waiting for initial osdmap 2024-04-26T09:35:36.455 INFO:tasks.ceph.osd.1.smithi194.stderr:2024-04-26T09:35:36.450+0000 7f4658aff700 -1 osd.1 0 waiting for initial osdmap 2024-04-26T09:35:36.455 INFO:tasks.ceph.osd.2.smithi194.stderr:2024-04-26T09:35:36.450+0000 7fee7014f700 -1 osd.2 0 waiting for initial osdmap 2024-04-26T09:35:36.459 INFO:tasks.ceph.osd.0.smithi194.stderr:2024-04-26T09:35:36.455+0000 7f73a1fd1700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T09:35:36.459 INFO:tasks.ceph.osd.2.smithi194.stderr:2024-04-26T09:35:36.455+0000 7fee6c8cb700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T09:35:36.459 INFO:tasks.ceph.osd.3.smithi194.stderr:2024-04-26T09:35:36.455+0000 7f658b92b700 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T09:35:36.459 INFO:tasks.ceph.osd.1.smithi194.stderr:2024-04-26T09:35:36.455+0000 7f465527b700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T09:35:37.727 DEBUG:teuthology.orchestra.run.smithi194:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T09:35:37.842 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.833+0000 7fda5c76d700 1 Processor -- start 2024-04-26T09:35:37.844 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.833+0000 7fda5c76d700 1 -- start start 2024-04-26T09:35:37.844 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5c76d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fda54122490 0x7fda54122870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:37.844 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5c76d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda54122d60 0x7fda5408c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:37.844 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5c76d700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5408e9f0 con 0x7fda5408c870 2024-04-26T09:35:37.844 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5c76d700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5408ec40 con 0x7fda54122d60 2024-04-26T09:35:37.844 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5c76d700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fda54122290 con 0x7fda54122490 2024-04-26T09:35:37.844 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5a509700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fda54122490 0x7fda54122870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:37.844 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5ad0a700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fda5408c870 0x7fda54129550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:55168/0 (socket says 172.21.15.194:55168) 2024-04-26T09:35:37.844 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda59d08700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda54122d60 0x7fda5408c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:37.844 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5ad0a700 1 -- 172.21.15.194:0/3791803706 learned_addr learned my addr 172.21.15.194:0/3791803706 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:35:37.844 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda59507700 1 -- 172.21.15.194:0/3791803706 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787630003 0 0) 0x7fda5408e9f0 con 0x7fda5408c870 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5a509700 1 -- 172.21.15.194:0/3791803706 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda54122d60 msgr2=0x7fda5408c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5a509700 1 --2- 172.21.15.194:0/3791803706 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda54122d60 0x7fda5408c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5a509700 1 -- 172.21.15.194:0/3791803706 >> v1:172.21.15.194:6789/0 conn(0x7fda5408c870 legacy=0x7fda54129550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5a509700 1 -- 172.21.15.194:0/3791803706 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda54090a70 con 0x7fda54122490 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda59d08700 1 --2- 172.21.15.194:0/3791803706 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda54122d60 0x7fda5408c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda5a509700 1 --2- 172.21.15.194:0/3791803706 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fda54122490 0x7fda54122870 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fda44009760 tx=0x7fda44006f40).ready entity=mon.2 client_cookie=32917ddb11762979 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.834+0000 7fda59507700 1 -- 172.21.15.194:0/3791803706 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda44005880 con 0x7fda54122490 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.835+0000 7fda59507700 1 -- 172.21.15.194:0/3791803706 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda440059e0 con 0x7fda54122490 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.835+0000 7fda59507700 1 -- 172.21.15.194:0/3791803706 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda44005d60 con 0x7fda54122490 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.835+0000 7fda5c76d700 1 -- 172.21.15.194:0/3791803706 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fda54122490 msgr2=0x7fda54122870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.835+0000 7fda5c76d700 1 --2- 172.21.15.194:0/3791803706 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fda54122490 0x7fda54122870 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fda44009760 tx=0x7fda44006f40).stop 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.835+0000 7fda5c76d700 1 -- 172.21.15.194:0/3791803706 shutdown_connections 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.835+0000 7fda5c76d700 1 --2- 172.21.15.194:0/3791803706 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fda54122490 0x7fda54122870 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.835+0000 7fda5c76d700 1 --2- 172.21.15.194:0/3791803706 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda54122d60 0x7fda5408c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:37.845 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.835+0000 7fda5c76d700 1 -- 172.21.15.194:0/3791803706 >> 172.21.15.194:0/3791803706 conn(0x7fda540919a0 msgr2=0x7fda54091db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.835+0000 7fda5c76d700 1 -- 172.21.15.194:0/3791803706 shutdown_connections 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.835+0000 7fda5c76d700 1 -- 172.21.15.194:0/3791803706 wait complete. 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5c76d700 1 Processor -- start 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5c76d700 1 -- start start 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5c76d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda5408c870 0x7fda5411b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5c76d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda54122490 0x7fda5411bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5c76d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fda54122d60 0x7fda5411c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5c76d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5411f100 con 0x7fda54122490 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5c76d700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda5411f430 con 0x7fda5408c870 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5c76d700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fda5411ed30 con 0x7fda54122d60 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5a509700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda5408c870 0x7fda5411b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda59d08700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda54122490 0x7fda5411bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5a509700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda5408c870 0x7fda5411b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:60212/0 (socket says 172.21.15.194:60212) 2024-04-26T09:35:37.846 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5a509700 1 -- 172.21.15.194:0/4091358315 learned_addr learned my addr 172.21.15.194:0/4091358315 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.836+0000 7fda5ad0a700 1 --2- 172.21.15.194:0/4091358315 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fda54122d60 0x7fda5411c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.837+0000 7fda5a509700 1 -- 172.21.15.194:0/4091358315 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fda54122d60 msgr2=0x7fda5411c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.837+0000 7fda5a509700 1 --2- 172.21.15.194:0/4091358315 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fda54122d60 0x7fda5411c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.837+0000 7fda5a509700 1 -- 172.21.15.194:0/4091358315 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda54122490 msgr2=0x7fda5411bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.837+0000 7fda5a509700 1 --2- 172.21.15.194:0/4091358315 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda54122490 0x7fda5411bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.837+0000 7fda5a509700 1 -- 172.21.15.194:0/4091358315 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda500092a0 con 0x7fda5408c870 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.837+0000 7fda5a509700 1 --2- 172.21.15.194:0/4091358315 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda5408c870 0x7fda5411b9d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fda541223d0 tx=0x7fda44015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.837+0000 7fda4affd700 1 -- 172.21.15.194:0/4091358315 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda44006750 con 0x7fda5408c870 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.837+0000 7fda4affd700 1 -- 172.21.15.194:0/4091358315 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda440068d0 con 0x7fda5408c870 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.837+0000 7fda5c76d700 1 -- 172.21.15.194:0/4091358315 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda44009250 con 0x7fda5408c870 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.837+0000 7fda4affd700 1 -- 172.21.15.194:0/4091358315 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4400f760 con 0x7fda5408c870 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.837+0000 7fda5c76d700 1 -- 172.21.15.194:0/4091358315 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5411d390 con 0x7fda5408c870 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.838+0000 7fda5c76d700 1 -- 172.21.15.194:0/4091358315 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda540631c0 con 0x7fda5408c870 2024-04-26T09:35:37.847 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.838+0000 7fda4affd700 1 -- 172.21.15.194:0/4091358315 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fda4400fb10 con 0x7fda5408c870 2024-04-26T09:35:37.848 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.838+0000 7fda4affd700 1 -- 172.21.15.194:0/4091358315 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 2069+0+0 (secure 0 0 0) 0x7fda44055d50 con 0x7fda5408c870 2024-04-26T09:35:37.848 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.842+0000 7fda4affd700 1 -- 172.21.15.194:0/4091358315 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fda44059030 con 0x7fda5408c870 2024-04-26T09:35:37.985 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.982+0000 7fda5c76d700 1 -- 172.21.15.194:0/4091358315 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda54046ce0 con 0x7fda5408c870 2024-04-26T09:35:37.986 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.982+0000 7fda4affd700 1 -- 172.21.15.194:0/4091358315 <== mon.1 v2:172.21.15.194:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v9) v1 ==== 74+0+4936 (secure 0 0 0) 0x7fda44022090 con 0x7fda5408c870 2024-04-26T09:35:37.986 INFO:teuthology.misc.health.smithi194.stdout: 2024-04-26T09:35:37.986 INFO:teuthology.misc.health.smithi194.stdout:{"epoch":9,"fsid":"be7f6752-916a-4f92-bdb5-9db81c891e5c","created":"2024-04-26T09:35:29.120276+0000","modified":"2024-04-26T09:35:37.446610+0000","last_up_change":"2024-04-26T09:35:37.446610+0000","last_in_change":"2024-04-26T09:35:30.442422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"79a6c3fb-32e2-46b3-81f7-7ab1e3bf3e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":72865}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":72865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":72865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":72865}]},"public_addr":"172.21.15.194:6808/72865","cluster_addr":"172.21.15.194:6809/72865","heartbeat_back_addr":"172.21.15.194:6811/72865","heartbeat_front_addr":"172.21.15.194:6810/72865","state":["exists","up"]},{"osd":1,"uuid":"be81777d-ab3b-46b3-873e-089f690a2249","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":72864}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":72864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":72864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":72864}]},"public_addr":"172.21.15.194:6800/72864","cluster_addr":"172.21.15.194:6801/72864","heartbeat_back_addr":"172.21.15.194:6803/72864","heartbeat_front_addr":"172.21.15.194:6802/72864","state":["exists","up"]},{"osd":2,"uuid":"efddf30f-dc2a-46d5-81d7-0a5cf1b01735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":72901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":72901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":72901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":72901}]},"public_addr":"172.21.15.194:6812/72901","cluster_addr":"172.21.15.194:6813/72901","heartbeat_back_addr":"172.21.15.194:6815/72901","heartbeat_front_addr":"172.21.15.194:6814/72901","state":["exists","up"]},{"osd":3,"uuid":"488b4848-1a54-4d45-b95d-ff5ccc61af12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":72866}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":72866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":72866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":72866}]},"public_addr":"172.21.15.194:6804/72866","cluster_addr":"172.21.15.194:6805/72866","heartbeat_back_addr":"172.21.15.194:6807/72866","heartbeat_front_addr":"172.21.15.194:6806/72866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T09:35:37.988 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.984+0000 7fda5c76d700 1 -- 172.21.15.194:0/4091358315 >> v1:172.21.15.194:6816/72244 conn(0x7fda38041b80 legacy=0x7fda38044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:37.988 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.984+0000 7fda5c76d700 1 -- 172.21.15.194:0/4091358315 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda5408c870 msgr2=0x7fda5411b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:37.988 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.984+0000 7fda5c76d700 1 --2- 172.21.15.194:0/4091358315 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda5408c870 0x7fda5411b9d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fda541223d0 tx=0x7fda44015000).stop 2024-04-26T09:35:37.988 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.984+0000 7fda5c76d700 1 -- 172.21.15.194:0/4091358315 shutdown_connections 2024-04-26T09:35:37.988 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.984+0000 7fda5c76d700 1 --2- 172.21.15.194:0/4091358315 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fda54122d60 0x7fda5411c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:37.988 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.984+0000 7fda5c76d700 1 --2- 172.21.15.194:0/4091358315 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fda5408c870 0x7fda5411b9d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:37.988 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.984+0000 7fda5c76d700 1 --2- 172.21.15.194:0/4091358315 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda54122490 0x7fda5411bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:37.988 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.984+0000 7fda5c76d700 1 -- 172.21.15.194:0/4091358315 >> 172.21.15.194:0/4091358315 conn(0x7fda540919a0 msgr2=0x7fda54065be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:37.988 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.984+0000 7fda5c76d700 1 -- 172.21.15.194:0/4091358315 shutdown_connections 2024-04-26T09:35:37.988 INFO:teuthology.misc.health.smithi194.stderr:2024-04-26T09:35:37.984+0000 7fda5c76d700 1 -- 172.21.15.194:0/4091358315 wait complete. 2024-04-26T09:35:37.998 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-04-26T09:35:37.998 INFO:tasks.ceph:Creating RBD pool 2024-04-26T09:35:37.998 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-26T09:35:38.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.108+0000 7f73350e1700 1 Processor -- start 2024-04-26T09:35:38.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.108+0000 7f73350e1700 1 -- start start 2024-04-26T09:35:38.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.108+0000 7f73350e1700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f733011ff10 0x7f73301202f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:38.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.108+0000 7f73350e1700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7330122fa0 0x7f7330123400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:38.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.108+0000 7f73350e1700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7330068fc0 con 0x7f73301238f0 2024-04-26T09:35:38.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.108+0000 7f73350e1700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7330123da0 con 0x7f733011ff10 2024-04-26T09:35:38.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.108+0000 7f73350e1700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f733011fcf0 con 0x7f7330122fa0 2024-04-26T09:35:38.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732e59c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7330122fa0 0x7f7330123400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:38.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732ed9d700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f733011ff10 0x7f73301202f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:38.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732e59c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7330122fa0 0x7f7330123400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46910/0 (socket says 172.21.15.194:46910) 2024-04-26T09:35:38.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732e59c700 1 -- 172.21.15.194:0/1621880028 learned_addr learned my addr 172.21.15.194:0/1621880028 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732f59e700 1 --1- 172.21.15.194:0/1621880028 >> v1:172.21.15.194:6789/0 conn(0x7f73301238f0 0x7f733012a340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:55172/0 (socket says 172.21.15.194:55172) 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732ed9d700 1 -- 172.21.15.194:0/1621880028 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7330122fa0 msgr2=0x7f7330123400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732ed9d700 1 --2- 172.21.15.194:0/1621880028 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7330122fa0 0x7f7330123400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732ed9d700 1 -- 172.21.15.194:0/1621880028 >> v1:172.21.15.194:6789/0 conn(0x7f73301238f0 legacy=0x7f733012a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732dd9b700 1 -- 172.21.15.194:0/1621880028 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063996346 0 0) 0x7f7328001490 con 0x7f73301238f0 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732ed9d700 1 -- 172.21.15.194:0/1621880028 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7330063ac0 con 0x7f733011ff10 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732e59c700 1 --2- 172.21.15.194:0/1621880028 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7330122fa0 0x7f7330123400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732ed9d700 1 --2- 172.21.15.194:0/1621880028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f733011ff10 0x7f73301202f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f7324009760 tx=0x7f7324009060).ready entity=mon.0 client_cookie=c638161fff8dab0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732dd9b700 1 -- 172.21.15.194:0/1621880028 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73240090d0 con 0x7f733011ff10 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732dd9b700 1 -- 172.21.15.194:0/1621880028 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732400f510 con 0x7f733011ff10 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.109+0000 7f732dd9b700 1 -- 172.21.15.194:0/1621880028 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732400f890 con 0x7f733011ff10 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.110+0000 7f73350e1700 1 -- 172.21.15.194:0/1621880028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f733011ff10 msgr2=0x7f73301202f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.110+0000 7f73350e1700 1 --2- 172.21.15.194:0/1621880028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f733011ff10 0x7f73301202f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f7324009760 tx=0x7f7324009060).stop 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.110+0000 7f73350e1700 1 -- 172.21.15.194:0/1621880028 shutdown_connections 2024-04-26T09:35:38.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.110+0000 7f73350e1700 1 --2- 172.21.15.194:0/1621880028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f733011ff10 0x7f73301202f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.110+0000 7f73350e1700 1 --2- 172.21.15.194:0/1621880028 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7330122fa0 0x7f7330123400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.110+0000 7f73350e1700 1 -- 172.21.15.194:0/1621880028 >> 172.21.15.194:0/1621880028 conn(0x7f7330095710 msgr2=0x7f7330095b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:38.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.110+0000 7f73350e1700 1 -- 172.21.15.194:0/1621880028 shutdown_connections 2024-04-26T09:35:38.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.110+0000 7f73350e1700 1 -- 172.21.15.194:0/1621880028 wait complete. 2024-04-26T09:35:38.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f73350e1700 1 Processor -- start 2024-04-26T09:35:38.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f73350e1700 1 -- start start 2024-04-26T09:35:38.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f73350e1700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f733011ff10 0x7f7330110570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:38.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f73350e1700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7330122fa0 0x7f7330110a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:38.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f73350e1700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f73301238f0 0x7f7330110f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:38.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f73350e1700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73301196d0 con 0x7f7330122fa0 2024-04-26T09:35:38.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f73350e1700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7330111440 con 0x7f733011ff10 2024-04-26T09:35:38.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f73350e1700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f73301190c0 con 0x7f73301238f0 2024-04-26T09:35:38.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f732f59e700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f73301238f0 0x7f7330110f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f732e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7330122fa0 0x7f7330110a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f732ed9d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f733011ff10 0x7f7330110570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f732e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7330122fa0 0x7f7330110a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55128/0 (socket says 172.21.15.194:55128) 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f732ed9d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f733011ff10 0x7f7330110570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:60224/0 (socket says 172.21.15.194:60224) 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f732e59c700 1 -- 172.21.15.194:0/1732956971 learned_addr learned my addr 172.21.15.194:0/1732956971 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.111+0000 7f732f59e700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f73301238f0 0x7f7330110f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46922/0 (socket says 172.21.15.194:46922) 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f732e59c700 1 -- 172.21.15.194:0/1732956971 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f73301238f0 msgr2=0x7f7330110f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f732e59c700 1 --2- 172.21.15.194:0/1732956971 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f73301238f0 0x7f7330110f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f732e59c700 1 -- 172.21.15.194:0/1732956971 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f733011ff10 msgr2=0x7f7330110570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f732e59c700 1 --2- 172.21.15.194:0/1732956971 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f733011ff10 0x7f7330110570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f732e59c700 1 -- 172.21.15.194:0/1732956971 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7320009370 con 0x7f7330122fa0 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f732f59e700 1 --2- 172.21.15.194:0/1732956971 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f73301238f0 0x7f7330110f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f732e59c700 1 --2- 172.21.15.194:0/1732956971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7330122fa0 0x7f7330110a60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f732000ea50 tx=0x7f73200090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f73177fe700 1 -- 172.21.15.194:0/1732956971 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7320009650 con 0x7f7330122fa0 2024-04-26T09:35:38.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f73177fe700 1 -- 172.21.15.194:0/1732956971 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f732000ee90 con 0x7f7330122fa0 2024-04-26T09:35:38.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f73177fe700 1 -- 172.21.15.194:0/1732956971 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732000ca00 con 0x7f7330122fa0 2024-04-26T09:35:38.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f73350e1700 1 -- 172.21.15.194:0/1732956971 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73240092a0 con 0x7f7330122fa0 2024-04-26T09:35:38.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.112+0000 7f73350e1700 1 -- 172.21.15.194:0/1732956971 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7330119370 con 0x7f7330122fa0 2024-04-26T09:35:38.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.113+0000 7f73177fe700 1 -- 172.21.15.194:0/1732956971 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f732001b400 con 0x7f7330122fa0 2024-04-26T09:35:38.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.113+0000 7f73177fe700 1 -- 172.21.15.194:0/1732956971 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 2069+0+0 (secure 0 0 0) 0x7f7320055180 con 0x7f7330122fa0 2024-04-26T09:35:38.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.113+0000 7f73350e1700 1 -- 172.21.15.194:0/1732956971 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7330119370 con 0x7f7330122fa0 2024-04-26T09:35:38.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.117+0000 7f73177fe700 1 -- 172.21.15.194:0/1732956971 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f7330119370 con 0x7f7330122fa0 2024-04-26T09:35:38.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.256+0000 7f73350e1700 1 -- 172.21.15.194:0/1732956971 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f7330119370 con 0x7f7330122fa0 2024-04-26T09:35:38.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.462+0000 7f73177fe700 1 -- 172.21.15.194:0/1732956971 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v10) v1 ==== 109+0+0 (secure 0 0 0) 0x7f7330119370 con 0x7f7330122fa0 2024-04-26T09:35:38.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.464+0000 7f73350e1700 1 -- 172.21.15.194:0/1732956971 >> v1:172.21.15.194:6816/72244 conn(0x7f731c041b80 legacy=0x7f731c044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:38.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.464+0000 7f73350e1700 1 -- 172.21.15.194:0/1732956971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7330122fa0 msgr2=0x7f7330110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:38.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.464+0000 7f73350e1700 1 --2- 172.21.15.194:0/1732956971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7330122fa0 0x7f7330110a60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f732000ea50 tx=0x7f73200090f0).stop 2024-04-26T09:35:38.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.464+0000 7f73350e1700 1 -- 172.21.15.194:0/1732956971 shutdown_connections 2024-04-26T09:35:38.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.464+0000 7f73350e1700 1 --2- 172.21.15.194:0/1732956971 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f73301238f0 0x7f7330110f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.464+0000 7f73350e1700 1 --2- 172.21.15.194:0/1732956971 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f733011ff10 0x7f7330110570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.464+0000 7f73350e1700 1 --2- 172.21.15.194:0/1732956971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7330122fa0 0x7f7330110a60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.464+0000 7f73350e1700 1 -- 172.21.15.194:0/1732956971 >> 172.21.15.194:0/1732956971 conn(0x7f7330095710 msgr2=0x7f733011f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:38.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.464+0000 7f73350e1700 1 -- 172.21.15.194:0/1732956971 shutdown_connections 2024-04-26T09:35:38.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.464+0000 7f73350e1700 1 -- 172.21.15.194:0/1732956971 wait complete. 2024-04-26T09:35:38.471 INFO:teuthology.orchestra.run.smithi194.stderr:pool 'rbd' created 2024-04-26T09:35:38.480 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-26T09:35:38.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.592+0000 7f6561a8d700 1 Processor -- start 2024-04-26T09:35:38.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.592+0000 7f6561a8d700 1 -- start start 2024-04-26T09:35:38.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.592+0000 7f6561a8d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 0x7f655c067250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:38.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.592+0000 7f6561a8d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c067740 0x7f655c11c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:38.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.592+0000 7f6561a8d700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f655c11c5c0 0x7f655c11ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:38.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.592+0000 7f6561a8d700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f655c11d180 con 0x7f655c11c5c0 2024-04-26T09:35:38.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.592+0000 7f6561a8d700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655c065090 con 0x7f655c067740 2024-04-26T09:35:38.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.592+0000 7f6561a8d700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f655c11cf60 con 0x7f655c066e70 2024-04-26T09:35:38.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655bfff700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f655c11c5c0 0x7f655c11ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:38.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655affd700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c067740 0x7f655c11c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:38.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655b7fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 0x7f655c067250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:38.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655bfff700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f655c11c5c0 0x7f655c11ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55134/0 (socket says 172.21.15.194:55134) 2024-04-26T09:35:38.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655affd700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c067740 0x7f655c11c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:60228/0 (socket says 172.21.15.194:60228) 2024-04-26T09:35:38.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655bfff700 1 -- 172.21.15.194:0/3677102919 learned_addr learned my addr 172.21.15.194:0/3677102919 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:38.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655affd700 1 -- 172.21.15.194:0/3677102919 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 msgr2=0x7f655c067250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:38.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655affd700 1 --2- 172.21.15.194:0/3677102919 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 0x7f655c067250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655affd700 1 -- 172.21.15.194:0/3677102919 >> v2:172.21.15.194:3300/0 conn(0x7f655c11c5c0 msgr2=0x7f655c11ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:38.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655affd700 1 --2- 172.21.15.194:0/3677102919 >> v2:172.21.15.194:3300/0 conn(0x7f655c11c5c0 0x7f655c11ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655affd700 1 -- 172.21.15.194:0/3677102919 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655c0688f0 con 0x7f655c067740 2024-04-26T09:35:38.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655bfff700 1 --2- 172.21.15.194:0/3677102919 >> v2:172.21.15.194:3300/0 conn(0x7f655c11c5c0 0x7f655c11ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:38.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655b7fe700 1 --2- 172.21.15.194:0/3677102919 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 0x7f655c067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:38.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655affd700 1 --2- 172.21.15.194:0/3677102919 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c067740 0x7f655c11c0d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f6544009790 tx=0x7f6544009060).ready entity=mon.1 client_cookie=e7f427bbfcec7068 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:38.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655a7fc700 1 -- 172.21.15.194:0/3677102919 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654400d4a0 con 0x7f655c067740 2024-04-26T09:35:38.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655a7fc700 1 -- 172.21.15.194:0/3677102919 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654400d600 con 0x7f655c067740 2024-04-26T09:35:38.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.593+0000 7f655a7fc700 1 -- 172.21.15.194:0/3677102919 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654400d980 con 0x7f655c067740 2024-04-26T09:35:38.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.594+0000 7f6561a8d700 1 -- 172.21.15.194:0/3677102919 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c067740 msgr2=0x7f655c11c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:38.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.594+0000 7f6561a8d700 1 --2- 172.21.15.194:0/3677102919 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c067740 0x7f655c11c0d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f6544009790 tx=0x7f6544009060).stop 2024-04-26T09:35:38.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.594+0000 7f6561a8d700 1 -- 172.21.15.194:0/3677102919 shutdown_connections 2024-04-26T09:35:38.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.594+0000 7f6561a8d700 1 --2- 172.21.15.194:0/3677102919 >> v2:172.21.15.194:3300/0 conn(0x7f655c11c5c0 0x7f655c11ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.594+0000 7f6561a8d700 1 --2- 172.21.15.194:0/3677102919 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 0x7f655c067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.594+0000 7f6561a8d700 1 --2- 172.21.15.194:0/3677102919 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c067740 0x7f655c11c0d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.594+0000 7f6561a8d700 1 -- 172.21.15.194:0/3677102919 >> 172.21.15.194:0/3677102919 conn(0x7f655c091bc0 msgr2=0x7f655c091fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:38.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.594+0000 7f6561a8d700 1 -- 172.21.15.194:0/3677102919 shutdown_connections 2024-04-26T09:35:38.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.594+0000 7f6561a8d700 1 -- 172.21.15.194:0/3677102919 wait complete. 2024-04-26T09:35:38.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.595+0000 7f6561a8d700 1 Processor -- start 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f6561a8d700 1 -- start start 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f6561a8d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 0x7f655c1352e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f6561a8d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f655c067740 0x7f655c1357d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f6561a8d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c11c5c0 0x7f655c135cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f6561a8d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c1361b0 con 0x7f655c067740 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f6561a8d700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655c1113b0 con 0x7f655c11c5c0 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f6561a8d700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f655c113700 con 0x7f655c066e70 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655b7fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 0x7f655c1352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f655c067740 0x7f655c1357d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655bfff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c11c5c0 0x7f655c135cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655b7fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 0x7f655c1352e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46944/0 (socket says 172.21.15.194:46944) 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f655c067740 0x7f655c1357d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55138/0 (socket says 172.21.15.194:55138) 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655b7fe700 1 -- 172.21.15.194:0/2838243746 learned_addr learned my addr 172.21.15.194:0/2838243746 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655bfff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c11c5c0 0x7f655c135cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:60236/0 (socket says 172.21.15.194:60236) 2024-04-26T09:35:38.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655b7fe700 1 -- 172.21.15.194:0/2838243746 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c11c5c0 msgr2=0x7f655c135cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:38.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655b7fe700 1 --2- 172.21.15.194:0/2838243746 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c11c5c0 0x7f655c135cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655b7fe700 1 -- 172.21.15.194:0/2838243746 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f655c067740 msgr2=0x7f655c1357d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:38.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655b7fe700 1 --2- 172.21.15.194:0/2838243746 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f655c067740 0x7f655c1357d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:38.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655b7fe700 1 -- 172.21.15.194:0/2838243746 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654c009a20 con 0x7f655c066e70 2024-04-26T09:35:38.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.596+0000 7f655bfff700 1 --2- 172.21.15.194:0/2838243746 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c11c5c0 0x7f655c135cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:38.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.597+0000 7f655b7fe700 1 --2- 172.21.15.194:0/2838243746 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 0x7f655c1352e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f654c00c1f0 tx=0x7f654c00b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:38.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.597+0000 7f6553fff700 1 -- 172.21.15.194:0/2838243746 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c009dd0 con 0x7f655c066e70 2024-04-26T09:35:38.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.597+0000 7f6553fff700 1 -- 172.21.15.194:0/2838243746 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654c00f040 con 0x7f655c066e70 2024-04-26T09:35:38.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.597+0000 7f6553fff700 1 -- 172.21.15.194:0/2838243746 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c00cee0 con 0x7f655c066e70 2024-04-26T09:35:38.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.597+0000 7f6561a8d700 1 -- 172.21.15.194:0/2838243746 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6554009370 con 0x7f655c066e70 2024-04-26T09:35:38.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.597+0000 7f6561a8d700 1 -- 172.21.15.194:0/2838243746 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65440092a0 con 0x7f655c066e70 2024-04-26T09:35:38.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.598+0000 7f6553fff700 1 -- 172.21.15.194:0/2838243746 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f654c013610 con 0x7f655c066e70 2024-04-26T09:35:38.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.598+0000 7f6561a8d700 1 -- 172.21.15.194:0/2838243746 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f655c067250 con 0x7f655c066e70 2024-04-26T09:35:38.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.598+0000 7f6553fff700 1 -- 172.21.15.194:0/2838243746 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 2803+0+0 (secure 0 0 0) 0x7f654c055ac0 con 0x7f655c066e70 2024-04-26T09:35:38.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.602+0000 7f6553fff700 1 -- 172.21.15.194:0/2838243746 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f654c036650 con 0x7f655c066e70 2024-04-26T09:35:38.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:38.741+0000 7f6561a8d700 1 -- 172.21.15.194:0/2838243746 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f655c1131c0 con 0x7f655c066e70 2024-04-26T09:35:39.472 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.469+0000 7f6553fff700 1 -- 172.21.15.194:0/2838243746 <== mon.2 v2:172.21.15.194:3302/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v11) v1 ==== 173+0+0 (secure 0 0 0) 0x7f654c055e60 con 0x7f655c066e70 2024-04-26T09:35:39.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.472+0000 7f6561a8d700 1 -- 172.21.15.194:0/2838243746 >> v1:172.21.15.194:6816/72244 conn(0x7f653c041b80 legacy=0x7f653c044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:39.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.472+0000 7f6561a8d700 1 -- 172.21.15.194:0/2838243746 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 msgr2=0x7f655c1352e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:39.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.472+0000 7f6561a8d700 1 --2- 172.21.15.194:0/2838243746 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 0x7f655c1352e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f654c00c1f0 tx=0x7f654c00b040).stop 2024-04-26T09:35:39.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.472+0000 7f6561a8d700 1 -- 172.21.15.194:0/2838243746 shutdown_connections 2024-04-26T09:35:39.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.472+0000 7f6561a8d700 1 --2- 172.21.15.194:0/2838243746 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f655c11c5c0 0x7f655c135cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.472+0000 7f6561a8d700 1 --2- 172.21.15.194:0/2838243746 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f655c066e70 0x7f655c1352e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.472+0000 7f6561a8d700 1 --2- 172.21.15.194:0/2838243746 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f655c067740 0x7f655c1357d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.472+0000 7f6561a8d700 1 -- 172.21.15.194:0/2838243746 >> 172.21.15.194:0/2838243746 conn(0x7f655c091bc0 msgr2=0x7f655c125f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:39.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.472+0000 7f6561a8d700 1 -- 172.21.15.194:0/2838243746 shutdown_connections 2024-04-26T09:35:39.477 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.472+0000 7f6561a8d700 1 -- 172.21.15.194:0/2838243746 wait complete. 2024-04-26T09:35:39.477 INFO:teuthology.orchestra.run.smithi194.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-26T09:35:39.491 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-26T09:35:39.492 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T09:35:39.492 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-26T09:35:39.492 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-26T09:35:39.493 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-26T09:35:39.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.605+0000 7fa517c0d700 1 Processor -- start 2024-04-26T09:35:39.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.605+0000 7fa517c0d700 1 -- start start 2024-04-26T09:35:39.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.605+0000 7fa517c0d700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fa510125670 0x7fa510125a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:39.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.605+0000 7fa517c0d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125f40 0x7fa510065c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:39.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.605+0000 7fa517c0d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510066570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.605+0000 7fa517c0d700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5100697b0 con 0x7fa510125670 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.605+0000 7fa517c0d700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100673b0 con 0x7fa510125f40 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.605+0000 7fa517c0d700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa5101252b0 con 0x7fa510066110 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5159a9700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fa510125670 0x7fa510125a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5151a8700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125f40 0x7fa510065c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5161aa700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510066570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5159a9700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fa510125670 0x7fa510125a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55148/0 (socket says 172.21.15.194:55148) 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5161aa700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510066570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46948/0 (socket says 172.21.15.194:46948) 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5151a8700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125f40 0x7fa510065c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:60252/0 (socket says 172.21.15.194:60252) 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5159a9700 1 -- 172.21.15.194:0/1169560406 learned_addr learned my addr 172.21.15.194:0/1169560406 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5151a8700 1 -- 172.21.15.194:0/1169560406 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 msgr2=0x7fa510066570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5151a8700 1 --2- 172.21.15.194:0/1169560406 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510066570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5151a8700 1 -- 172.21.15.194:0/1169560406 >> v2:172.21.15.194:3300/0 conn(0x7fa510125670 msgr2=0x7fa510125a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:39.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5151a8700 1 --2- 172.21.15.194:0/1169560406 >> v2:172.21.15.194:3300/0 conn(0x7fa510125670 0x7fa510125a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5151a8700 1 -- 172.21.15.194:0/1169560406 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa510066a60 con 0x7fa510125f40 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5161aa700 1 --2- 172.21.15.194:0/1169560406 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510066570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5159a9700 1 --2- 172.21.15.194:0/1169560406 >> v2:172.21.15.194:3300/0 conn(0x7fa510125670 0x7fa510125a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5151a8700 1 --2- 172.21.15.194:0/1169560406 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125f40 0x7fa510065c20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fa500009790 tx=0x7fa500009060).ready entity=mon.1 client_cookie=42c98cc9714ed834 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5149a7700 1 -- 172.21.15.194:0/1169560406 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5000090d0 con 0x7fa510125f40 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5149a7700 1 -- 172.21.15.194:0/1169560406 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50000d4a0 con 0x7fa510125f40 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.606+0000 7fa5149a7700 1 -- 172.21.15.194:0/1169560406 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50000d820 con 0x7fa510125f40 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.607+0000 7fa517c0d700 1 -- 172.21.15.194:0/1169560406 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125f40 msgr2=0x7fa510065c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.607+0000 7fa517c0d700 1 --2- 172.21.15.194:0/1169560406 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125f40 0x7fa510065c20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fa500009790 tx=0x7fa500009060).stop 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.607+0000 7fa517c0d700 1 -- 172.21.15.194:0/1169560406 shutdown_connections 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.607+0000 7fa517c0d700 1 --2- 172.21.15.194:0/1169560406 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510066570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.607+0000 7fa517c0d700 1 --2- 172.21.15.194:0/1169560406 >> v2:172.21.15.194:3300/0 conn(0x7fa510125670 0x7fa510125a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.607+0000 7fa517c0d700 1 --2- 172.21.15.194:0/1169560406 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125f40 0x7fa510065c20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.607+0000 7fa517c0d700 1 -- 172.21.15.194:0/1169560406 >> 172.21.15.194:0/1169560406 conn(0x7fa5100957b0 msgr2=0x7fa510095bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.607+0000 7fa517c0d700 1 -- 172.21.15.194:0/1169560406 shutdown_connections 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.607+0000 7fa517c0d700 1 -- 172.21.15.194:0/1169560406 wait complete. 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa517c0d700 1 Processor -- start 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa517c0d700 1 -- start start 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa517c0d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510117200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa517c0d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125670 0x7fa5101176f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa517c0d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa510125f40 0x7fa510117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa517c0d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51011a720 con 0x7fa510125f40 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa517c0d700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa510118df0 con 0x7fa510125670 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa517c0d700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa510119f40 con 0x7fa510066110 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5159a9700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510117200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5151a8700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125670 0x7fa5101176f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5159a9700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510117200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46960/0 (socket says 172.21.15.194:46960) 2024-04-26T09:35:39.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5159a9700 1 -- 172.21.15.194:0/146252324 learned_addr learned my addr 172.21.15.194:0/146252324 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5161aa700 1 --2- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa510125f40 0x7fa510117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5159a9700 1 -- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125670 msgr2=0x7fa5101176f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5159a9700 1 --2- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125670 0x7fa5101176f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5159a9700 1 -- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa510125f40 msgr2=0x7fa510117be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5159a9700 1 --2- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa510125f40 0x7fa510117be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5159a9700 1 -- 172.21.15.194:0/146252324 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa508009a20 con 0x7fa510066110 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5151a8700 1 --2- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125670 0x7fa5101176f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.608+0000 7fa5161aa700 1 --2- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa510125f40 0x7fa510117be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.609+0000 7fa5159a9700 1 --2- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510117200 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa50c00ea20 tx=0x7fa50c0090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.609+0000 7fa5067fc700 1 -- 172.21.15.194:0/146252324 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c009650 con 0x7fa510066110 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.609+0000 7fa5067fc700 1 -- 172.21.15.194:0/146252324 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50c00c850 con 0x7fa510066110 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.609+0000 7fa517c0d700 1 -- 172.21.15.194:0/146252324 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50c009370 con 0x7fa510066110 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.609+0000 7fa5067fc700 1 -- 172.21.15.194:0/146252324 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50c00cbd0 con 0x7fa510066110 2024-04-26T09:35:39.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.609+0000 7fa517c0d700 1 -- 172.21.15.194:0/146252324 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5000092a0 con 0x7fa510066110 2024-04-26T09:35:39.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.610+0000 7fa5067fc700 1 -- 172.21.15.194:0/146252324 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fa50c01b400 con 0x7fa510066110 2024-04-26T09:35:39.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.610+0000 7fa5067fc700 1 -- 172.21.15.194:0/146252324 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fa50c055710 con 0x7fa510066110 2024-04-26T09:35:39.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.610+0000 7fa517c0d700 1 -- 172.21.15.194:0/146252324 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa510122c70 con 0x7fa510066110 2024-04-26T09:35:39.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.614+0000 7fa5067fc700 1 -- 172.21.15.194:0/146252324 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fa50c016080 con 0x7fa510066110 2024-04-26T09:35:39.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.781+0000 7fa517c0d700 1 -- 172.21.15.194:0/146252324 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fa510092a10 con 0x7fa510066110 2024-04-26T09:35:39.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.783+0000 7fa5067fc700 1 -- 172.21.15.194:0/146252324 <== mon.2 v2:172.21.15.194:3302/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+97489 (secure 0 0 0) 0x7fa50c036280 con 0x7fa510066110 2024-04-26T09:35:39.788 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:39.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.785+0000 7fa517c0d700 1 -- 172.21.15.194:0/146252324 >> v1:172.21.15.194:6816/72244 conn(0x7fa4f4041b80 legacy=0x7fa4f4044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:39.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.785+0000 7fa517c0d700 1 -- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 msgr2=0x7fa510117200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:39.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.785+0000 7fa517c0d700 1 --2- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510117200 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa50c00ea20 tx=0x7fa50c0090f0).stop 2024-04-26T09:35:39.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.785+0000 7fa517c0d700 1 -- 172.21.15.194:0/146252324 shutdown_connections 2024-04-26T09:35:39.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.785+0000 7fa517c0d700 1 --2- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa510125f40 0x7fa510117be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.785+0000 7fa517c0d700 1 --2- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa510066110 0x7fa510117200 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.785+0000 7fa517c0d700 1 --2- 172.21.15.194:0/146252324 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa510125670 0x7fa5101176f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.785+0000 7fa517c0d700 1 -- 172.21.15.194:0/146252324 >> 172.21.15.194:0/146252324 conn(0x7fa5100957b0 msgr2=0x7fa510093ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:39.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.785+0000 7fa517c0d700 1 -- 172.21.15.194:0/146252324 shutdown_connections 2024-04-26T09:35:39.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.785+0000 7fa517c0d700 1 -- 172.21.15.194:0/146252324 wait complete. 2024-04-26T09:35:39.800 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":4,"active_gid":4111,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6816","nonce":72244}]},"active_addr":"172.21.15.194:6816/72244","active_change":"2024-04-26T09:35:32.429536+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[{"gid":4105,"name":"y","mgr_features":4540138314316775423,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.194:0","nonce":3269138671}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.194:0","nonce":3797490727}]}]} 2024-04-26T09:35:39.805 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-26T09:35:39.805 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-26T09:35:39.805 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T09:35:39.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.916+0000 7f6c63f10700 1 Processor -- start 2024-04-26T09:35:39.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.916+0000 7f6c63f10700 1 -- start start 2024-04-26T09:35:39.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c63f10700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 0x7f6c5c066000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:39.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c63f10700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f6c5c1226a0 0x7f6c5c12a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:39.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c63f10700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c0695c0 con 0x7f6c5c0664f0 2024-04-26T09:35:39.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c63f10700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5c069810 con 0x7f6c5c1226a0 2024-04-26T09:35:39.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c63f10700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c069290 con 0x7f6c5c065c20 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c61cac700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 0x7f6c5c066000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c624ad700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f6c5c1226a0 0x7f6c5c12a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c61cac700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 0x7f6c5c066000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:60266/0 (socket says 172.21.15.194:60266) 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c61cac700 1 -- 172.21.15.194:0/2870907508 learned_addr learned my addr 172.21.15.194:0/2870907508 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c614ab700 1 --1- 172.21.15.194:0/2870907508 >> v1:172.21.15.194:6789/0 conn(0x7f6c5c0664f0 0x7f6c5c121f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:55182/0 (socket says 172.21.15.194:55182) 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c624ad700 1 -- 172.21.15.194:0/2870907508 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 msgr2=0x7f6c5c066000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c624ad700 1 --2- 172.21.15.194:0/2870907508 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 0x7f6c5c066000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c624ad700 1 -- 172.21.15.194:0/2870907508 >> v1:172.21.15.194:6789/0 conn(0x7f6c5c0664f0 legacy=0x7f6c5c121f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c60caa700 1 -- 172.21.15.194:0/2870907508 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267309378 0 0) 0x7f6c5c0695c0 con 0x7f6c5c0664f0 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c624ad700 1 -- 172.21.15.194:0/2870907508 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c0684c0 con 0x7f6c5c1226a0 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c624ad700 1 --2- 172.21.15.194:0/2870907508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c1226a0 0x7f6c5c12a340 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6c54009e60 tx=0x7f6c54009780).ready entity=mon.0 client_cookie=cdf3bbc6c18e3e9f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c60caa700 1 -- 172.21.15.194:0/2870907508 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c5400e070 con 0x7f6c5c1226a0 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c60caa700 1 -- 172.21.15.194:0/2870907508 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c5400bb60 con 0x7f6c5c1226a0 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.917+0000 7f6c60caa700 1 -- 172.21.15.194:0/2870907508 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54012410 con 0x7f6c5c1226a0 2024-04-26T09:35:39.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.918+0000 7f6c63f10700 1 -- 172.21.15.194:0/2870907508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c1226a0 msgr2=0x7f6c5c12a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:39.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.918+0000 7f6c63f10700 1 --2- 172.21.15.194:0/2870907508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c1226a0 0x7f6c5c12a340 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f6c54009e60 tx=0x7f6c54009780).stop 2024-04-26T09:35:39.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.918+0000 7f6c63f10700 1 -- 172.21.15.194:0/2870907508 shutdown_connections 2024-04-26T09:35:39.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.918+0000 7f6c63f10700 1 --2- 172.21.15.194:0/2870907508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c1226a0 0x7f6c5c12a340 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.918+0000 7f6c63f10700 1 --2- 172.21.15.194:0/2870907508 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 0x7f6c5c066000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.918+0000 7f6c63f10700 1 -- 172.21.15.194:0/2870907508 >> 172.21.15.194:0/2870907508 conn(0x7f6c5c0957b0 msgr2=0x7f6c5c095bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:39.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.918+0000 7f6c63f10700 1 -- 172.21.15.194:0/2870907508 shutdown_connections 2024-04-26T09:35:39.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.918+0000 7f6c63f10700 1 -- 172.21.15.194:0/2870907508 wait complete. 2024-04-26T09:35:39.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c63f10700 1 Processor -- start 2024-04-26T09:35:39.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c63f10700 1 -- start start 2024-04-26T09:35:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c63f10700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 0x7f6c5c113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c63f10700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c0664f0 0x7f6c5c113520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c63f10700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f6c5c1226a0 0x7f6c5c113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c63f10700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c114420 con 0x7f6c5c0664f0 2024-04-26T09:35:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c63f10700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c116890 con 0x7f6c5c065c20 2024-04-26T09:35:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c63f10700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c1164c0 con 0x7f6c5c1226a0 2024-04-26T09:35:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c61cac700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 0x7f6c5c113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c614ab700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c0664f0 0x7f6c5c113520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c61cac700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 0x7f6c5c113030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:60276/0 (socket says 172.21.15.194:60276) 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c61cac700 1 -- 172.21.15.194:0/1041088859 learned_addr learned my addr 172.21.15.194:0/1041088859 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.919+0000 7f6c624ad700 1 --2- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f6c5c1226a0 0x7f6c5c113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c614ab700 1 -- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f6c5c1226a0 msgr2=0x7f6c5c113a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c614ab700 1 --2- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f6c5c1226a0 0x7f6c5c113a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c614ab700 1 -- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 msgr2=0x7f6c5c113030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c614ab700 1 --2- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 0x7f6c5c113030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c614ab700 1 -- 172.21.15.194:0/1041088859 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c54009a20 con 0x7f6c5c0664f0 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c61cac700 1 --2- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 0x7f6c5c113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c614ab700 1 --2- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c0664f0 0x7f6c5c113520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f6c58000c00 tx=0x7f6c58007bb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c527fc700 1 -- 172.21.15.194:0/1041088859 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c580070d0 con 0x7f6c5c0664f0 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c527fc700 1 -- 172.21.15.194:0/1041088859 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c58007250 con 0x7f6c5c0664f0 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c527fc700 1 -- 172.21.15.194:0/1041088859 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c580056e0 con 0x7f6c5c0664f0 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c63f10700 1 -- 172.21.15.194:0/1041088859 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5c115e00 con 0x7f6c5c0664f0 2024-04-26T09:35:39.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.920+0000 7f6c63f10700 1 -- 172.21.15.194:0/1041088859 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5c113f80 con 0x7f6c5c0664f0 2024-04-26T09:35:39.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.921+0000 7f6c63f10700 1 -- 172.21.15.194:0/1041088859 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c116050 con 0x7f6c5c0664f0 2024-04-26T09:35:39.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.921+0000 7f6c527fc700 1 -- 172.21.15.194:0/1041088859 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f6c58005a50 con 0x7f6c5c0664f0 2024-04-26T09:35:39.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.922+0000 7f6c527fc700 1 -- 172.21.15.194:0/1041088859 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f6c58054d10 con 0x7f6c5c0664f0 2024-04-26T09:35:39.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:39.925+0000 7f6c527fc700 1 -- 172.21.15.194:0/1041088859 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f6c580357b0 con 0x7f6c5c0664f0 2024-04-26T09:35:40.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.064+0000 7f6c63f10700 1 -- 172.21.15.194:0/1041088859 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c5c115c40 con 0x7f6c5c0664f0 2024-04-26T09:35:40.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.065+0000 7f6c527fc700 1 -- 172.21.15.194:0/1041088859 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+8150 (secure 0 0 0) 0x7f6c5c115c40 con 0x7f6c5c0664f0 2024-04-26T09:35:40.068 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:40.069 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":11,"fsid":"be7f6752-916a-4f92-bdb5-9db81c891e5c","created":"2024-04-26T09:35:29.120276+0000","modified":"2024-04-26T09:35:39.459217+0000","last_up_change":"2024-04-26T09:35:37.446610+0000","last_in_change":"2024-04-26T09:35:30.442422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-04-26T09:35:37.489778+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T09:35:38.288163+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"79a6c3fb-32e2-46b3-81f7-7ab1e3bf3e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":72865}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":72865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":72865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":72865}]},"public_addr":"172.21.15.194:6808/72865","cluster_addr":"172.21.15.194:6809/72865","heartbeat_back_addr":"172.21.15.194:6811/72865","heartbeat_front_addr":"172.21.15.194:6810/72865","state":["exists","up"]},{"osd":1,"uuid":"be81777d-ab3b-46b3-873e-089f690a2249","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":72864}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":72864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":72864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":72864}]},"public_addr":"172.21.15.194:6800/72864","cluster_addr":"172.21.15.194:6801/72864","heartbeat_back_addr":"172.21.15.194:6803/72864","heartbeat_front_addr":"172.21.15.194:6802/72864","state":["exists","up"]},{"osd":2,"uuid":"efddf30f-dc2a-46d5-81d7-0a5cf1b01735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":72901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":72901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":72901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":72901}]},"public_addr":"172.21.15.194:6812/72901","cluster_addr":"172.21.15.194:6813/72901","heartbeat_back_addr":"172.21.15.194:6815/72901","heartbeat_front_addr":"172.21.15.194:6814/72901","state":["exists","up"]},{"osd":3,"uuid":"488b4848-1a54-4d45-b95d-ff5ccc61af12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":72866}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":72866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":72866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":72866}]},"public_addr":"172.21.15.194:6804/72866","cluster_addr":"172.21.15.194:6805/72866","heartbeat_back_addr":"172.21.15.194:6807/72866","heartbeat_front_addr":"172.21.15.194:6806/72866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.624174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.581378+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.599989+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.632377+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T09:35:40.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.066+0000 7f6c63f10700 1 -- 172.21.15.194:0/1041088859 >> v1:172.21.15.194:6816/72244 conn(0x7f6c40041b80 legacy=0x7f6c40044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.066+0000 7f6c63f10700 1 -- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c0664f0 msgr2=0x7f6c5c113520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.066+0000 7f6c63f10700 1 --2- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c0664f0 0x7f6c5c113520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f6c58000c00 tx=0x7f6c58007bb0).stop 2024-04-26T09:35:40.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.066+0000 7f6c63f10700 1 -- 172.21.15.194:0/1041088859 shutdown_connections 2024-04-26T09:35:40.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.066+0000 7f6c63f10700 1 --2- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f6c5c1226a0 0x7f6c5c113a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.066+0000 7f6c63f10700 1 --2- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6c5c065c20 0x7f6c5c113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.066+0000 7f6c63f10700 1 --2- 172.21.15.194:0/1041088859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c5c0664f0 0x7f6c5c113520 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.066+0000 7f6c63f10700 1 -- 172.21.15.194:0/1041088859 >> 172.21.15.194:0/1041088859 conn(0x7f6c5c0957b0 msgr2=0x7f6c5c0932a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.066+0000 7f6c63f10700 1 -- 172.21.15.194:0/1041088859 shutdown_connections 2024-04-26T09:35:40.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.066+0000 7f6c63f10700 1 -- 172.21.15.194:0/1041088859 wait complete. 2024-04-26T09:35:40.081 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-26T09:35:40.082 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T09:35:40.195 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9ebc234700 1 Processor -- start 2024-04-26T09:35:40.195 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9ebc234700 1 -- start start 2024-04-26T09:35:40.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9ebc234700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f9eb40696d0 0x7f9eb4069ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9ebc234700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 0x7f9eb40665b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9ebc234700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb4067f30 con 0x7f9eb4066aa0 2024-04-26T09:35:40.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9ebc234700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eb4090be0 con 0x7f9eb40696d0 2024-04-26T09:35:40.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9ebc234700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb405e8c0 con 0x7f9eb4069fa0 2024-04-26T09:35:40.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9eb97cf700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 0x7f9eb40665b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9eb9fd0700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f9eb40696d0 0x7f9eb4069ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9eba7d1700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f9eb4066aa0 0x7f9eb4066f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:55184/0 (socket says 172.21.15.194:55184) 2024-04-26T09:35:40.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9eb97cf700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 0x7f9eb40665b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46980/0 (socket says 172.21.15.194:46980) 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.192+0000 7f9eba7d1700 1 -- 172.21.15.194:0/2013486228 learned_addr learned my addr 172.21.15.194:0/2013486228 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9eb97cf700 1 -- 172.21.15.194:0/2013486228 >> v2:172.21.15.194:3300/0 conn(0x7f9eb40696d0 msgr2=0x7f9eb4069ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9eb97cf700 1 --2- 172.21.15.194:0/2013486228 >> v2:172.21.15.194:3300/0 conn(0x7f9eb40696d0 0x7f9eb4069ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9eb97cf700 1 -- 172.21.15.194:0/2013486228 >> v1:172.21.15.194:6789/0 conn(0x7f9eb4066aa0 legacy=0x7f9eb4066f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9eb97cf700 1 -- 172.21.15.194:0/2013486228 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb4068290 con 0x7f9eb4069fa0 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9eb8fce700 1 -- 172.21.15.194:0/2013486228 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370312795 0 0) 0x7f9eb4067f30 con 0x7f9eb4066aa0 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9eb9fd0700 1 --2- 172.21.15.194:0/2013486228 >> v2:172.21.15.194:3300/0 conn(0x7f9eb40696d0 0x7f9eb4069ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9eb97cf700 1 --2- 172.21.15.194:0/2013486228 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 0x7f9eb40665b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9ea4009760 tx=0x7f9ea4009060).ready entity=mon.2 client_cookie=dfe766a4226d4f36 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9eb8fce700 1 -- 172.21.15.194:0/2013486228 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea40090d0 con 0x7f9eb4069fa0 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9eb8fce700 1 -- 172.21.15.194:0/2013486228 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea400f510 con 0x7f9eb4069fa0 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9eb8fce700 1 -- 172.21.15.194:0/2013486228 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea400f890 con 0x7f9eb4069fa0 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9ebc234700 1 -- 172.21.15.194:0/2013486228 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 msgr2=0x7f9eb40665b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9ebc234700 1 --2- 172.21.15.194:0/2013486228 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 0x7f9eb40665b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9ea4009760 tx=0x7f9ea4009060).stop 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.193+0000 7f9ebc234700 1 -- 172.21.15.194:0/2013486228 shutdown_connections 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.194+0000 7f9ebc234700 1 --2- 172.21.15.194:0/2013486228 >> v2:172.21.15.194:3300/0 conn(0x7f9eb40696d0 0x7f9eb4069ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.194+0000 7f9ebc234700 1 --2- 172.21.15.194:0/2013486228 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 0x7f9eb40665b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.194+0000 7f9ebc234700 1 -- 172.21.15.194:0/2013486228 >> 172.21.15.194:0/2013486228 conn(0x7f9eb4091910 msgr2=0x7f9eb4091d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.194+0000 7f9ebc234700 1 -- 172.21.15.194:0/2013486228 shutdown_connections 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.194+0000 7f9ebc234700 1 -- 172.21.15.194:0/2013486228 wait complete. 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.194+0000 7f9ebc234700 1 Processor -- start 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.194+0000 7f9ebc234700 1 -- start start 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9ebc234700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9eb4066aa0 0x7f9eb4132aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9ebc234700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9eb40696d0 0x7f9eb4132f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9ebc234700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 0x7f9eb4133480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9ebc234700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb40830b0 con 0x7f9eb4066aa0 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9ebc234700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb4122fa0 con 0x7f9eb40696d0 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9ebc234700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb40847c0 con 0x7f9eb4069fa0 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eb9fd0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9eb4066aa0 0x7f9eb4132aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eb9fd0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9eb4066aa0 0x7f9eb4132aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55198/0 (socket says 172.21.15.194:55198) 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eba7d1700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 0x7f9eb4133480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eb97cf700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9eb40696d0 0x7f9eb4132f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eb9fd0700 1 -- 172.21.15.194:0/3718461552 learned_addr learned my addr 172.21.15.194:0/3718461552 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eb9fd0700 1 -- 172.21.15.194:0/3718461552 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 msgr2=0x7f9eb4133480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eb9fd0700 1 --2- 172.21.15.194:0/3718461552 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 0x7f9eb4133480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eb9fd0700 1 -- 172.21.15.194:0/3718461552 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9eb40696d0 msgr2=0x7f9eb4132f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eb9fd0700 1 --2- 172.21.15.194:0/3718461552 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9eb40696d0 0x7f9eb4132f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eb9fd0700 1 -- 172.21.15.194:0/3718461552 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb0009250 con 0x7f9eb4066aa0 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eba7d1700 1 --2- 172.21.15.194:0/3718461552 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 0x7f9eb4133480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eb9fd0700 1 --2- 172.21.15.194:0/3718461552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9eb4066aa0 0x7f9eb4132aa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9eb000ec80 tx=0x7f9eb0006f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eaa7fc700 1 -- 172.21.15.194:0/3718461552 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb00096a0 con 0x7f9eb4066aa0 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eaa7fc700 1 -- 172.21.15.194:0/3718461552 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb0005880 con 0x7f9eb4066aa0 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9eaa7fc700 1 -- 172.21.15.194:0/3718461552 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0005c00 con 0x7f9eb4066aa0 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.195+0000 7f9ebc234700 1 -- 172.21.15.194:0/3718461552 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea40092a0 con 0x7f9eb4066aa0 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.196+0000 7f9ebc234700 1 -- 172.21.15.194:0/3718461552 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb41225f0 con 0x7f9eb4066aa0 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.197+0000 7f9eaa7fc700 1 -- 172.21.15.194:0/3718461552 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f9eb000c850 con 0x7f9eb4066aa0 2024-04-26T09:35:40.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.197+0000 7f9ebc234700 1 -- 172.21.15.194:0/3718461552 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb4090440 con 0x7f9eb4066aa0 2024-04-26T09:35:40.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.197+0000 7f9eaa7fc700 1 -- 172.21.15.194:0/3718461552 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f9eb0055290 con 0x7f9eb4066aa0 2024-04-26T09:35:40.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.200+0000 7f9eaa7fc700 1 -- 172.21.15.194:0/3718461552 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f9eb0035c90 con 0x7f9eb4066aa0 2024-04-26T09:35:40.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.339+0000 7f9ebc234700 1 -- 172.21.15.194:0/3718461552 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9eb405e430 con 0x7f9eb4066aa0 2024-04-26T09:35:40.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.340+0000 7f9eaa7fc700 1 -- 172.21.15.194:0/3718461552 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+8150 (secure 0 0 0) 0x7f9eb0016030 con 0x7f9eb4066aa0 2024-04-26T09:35:40.344 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:40.344 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":11,"fsid":"be7f6752-916a-4f92-bdb5-9db81c891e5c","created":"2024-04-26T09:35:29.120276+0000","modified":"2024-04-26T09:35:39.459217+0000","last_up_change":"2024-04-26T09:35:37.446610+0000","last_in_change":"2024-04-26T09:35:30.442422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-04-26T09:35:37.489778+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T09:35:38.288163+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"79a6c3fb-32e2-46b3-81f7-7ab1e3bf3e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":72865}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":72865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":72865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":72865}]},"public_addr":"172.21.15.194:6808/72865","cluster_addr":"172.21.15.194:6809/72865","heartbeat_back_addr":"172.21.15.194:6811/72865","heartbeat_front_addr":"172.21.15.194:6810/72865","state":["exists","up"]},{"osd":1,"uuid":"be81777d-ab3b-46b3-873e-089f690a2249","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":72864}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":72864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":72864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":72864}]},"public_addr":"172.21.15.194:6800/72864","cluster_addr":"172.21.15.194:6801/72864","heartbeat_back_addr":"172.21.15.194:6803/72864","heartbeat_front_addr":"172.21.15.194:6802/72864","state":["exists","up"]},{"osd":2,"uuid":"efddf30f-dc2a-46d5-81d7-0a5cf1b01735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":72901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":72901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":72901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":72901}]},"public_addr":"172.21.15.194:6812/72901","cluster_addr":"172.21.15.194:6813/72901","heartbeat_back_addr":"172.21.15.194:6815/72901","heartbeat_front_addr":"172.21.15.194:6814/72901","state":["exists","up"]},{"osd":3,"uuid":"488b4848-1a54-4d45-b95d-ff5ccc61af12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":72866}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":72866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":72866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":72866}]},"public_addr":"172.21.15.194:6804/72866","cluster_addr":"172.21.15.194:6805/72866","heartbeat_back_addr":"172.21.15.194:6807/72866","heartbeat_front_addr":"172.21.15.194:6806/72866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.624174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.581378+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.599989+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.632377+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T09:35:40.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.341+0000 7f9ebc234700 1 -- 172.21.15.194:0/3718461552 >> v1:172.21.15.194:6816/72244 conn(0x7f9e98041b80 legacy=0x7f9e98044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.341+0000 7f9ebc234700 1 -- 172.21.15.194:0/3718461552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9eb4066aa0 msgr2=0x7f9eb4132aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.341+0000 7f9ebc234700 1 --2- 172.21.15.194:0/3718461552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9eb4066aa0 0x7f9eb4132aa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9eb000ec80 tx=0x7f9eb0006f40).stop 2024-04-26T09:35:40.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.341+0000 7f9ebc234700 1 -- 172.21.15.194:0/3718461552 shutdown_connections 2024-04-26T09:35:40.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.341+0000 7f9ebc234700 1 --2- 172.21.15.194:0/3718461552 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9eb4069fa0 0x7f9eb4133480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.341+0000 7f9ebc234700 1 --2- 172.21.15.194:0/3718461552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9eb4066aa0 0x7f9eb4132aa0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.341+0000 7f9ebc234700 1 --2- 172.21.15.194:0/3718461552 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9eb40696d0 0x7f9eb4132f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.341+0000 7f9ebc234700 1 -- 172.21.15.194:0/3718461552 >> 172.21.15.194:0/3718461552 conn(0x7f9eb4091910 msgr2=0x7f9eb4123620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.341+0000 7f9ebc234700 1 -- 172.21.15.194:0/3718461552 shutdown_connections 2024-04-26T09:35:40.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.341+0000 7f9ebc234700 1 -- 172.21.15.194:0/3718461552 wait complete. 2024-04-26T09:35:40.357 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T09:35:40.357 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-26T09:35:40.358 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T09:35:40.358 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T09:35:40.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.482+0000 7f7ef64c9700 1 Processor -- start 2024-04-26T09:35:40.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.482+0000 7f7ef64c9700 1 -- start start 2024-04-26T09:35:40.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.482+0000 7f0eb2987700 1 Processor -- start 2024-04-26T09:35:40.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.482+0000 7f0eb2987700 1 -- start start 2024-04-26T09:35:40.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.482+0000 7f7ef64c9700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f7ef0090df0 0x7f7ef008c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.482+0000 7f7ef64c9700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008c720 0x7f7ef008cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.482+0000 7f7ef64c9700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f7ef008b010 0x7f7ef008b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.482+0000 7f7ef64c9700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ef008edf0 con 0x7f7ef0090df0 2024-04-26T09:35:40.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.482+0000 7f7ef64c9700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef006cdd0 con 0x7f7ef008b010 2024-04-26T09:35:40.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.482+0000 7f7ef64c9700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef008f070 con 0x7f7ef008c720 2024-04-26T09:35:40.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb2987700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0eac090ad0 0x7f0eac090eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb2987700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac12abf0 0x7f0eac12cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.482+0000 7f7eef7fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008c720 0x7f7ef008cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f7eeffff700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f7ef0090df0 0x7f7ef008c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f7eef7fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008c720 0x7f7ef008cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46992/0 (socket says 172.21.15.194:46992) 2024-04-26T09:35:40.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f7eef7fe700 1 -- 172.21.15.194:0/2316264980 learned_addr learned my addr 172.21.15.194:0/2316264980 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb2987700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac08e9d0 con 0x7f0eac08eb60 2024-04-26T09:35:40.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb2987700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac066660 con 0x7f0eac090ad0 2024-04-26T09:35:40.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb2987700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac066a30 con 0x7f0eac12abf0 2024-04-26T09:35:40.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f7eef7fe700 1 -- 172.21.15.194:0/2316264980 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f7ef008b010 msgr2=0x7f7ef008b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f7eef7fe700 1 --2- 172.21.15.194:0/2316264980 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f7ef008b010 0x7f7ef008b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:40.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f7eef7fe700 1 -- 172.21.15.194:0/2316264980 >> v2:172.21.15.194:3300/0 conn(0x7f7ef0090df0 msgr2=0x7f7ef008c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f7eef7fe700 1 --2- 172.21.15.194:0/2316264980 >> v2:172.21.15.194:3300/0 conn(0x7f7ef0090df0 0x7f7ef008c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f7eef7fe700 1 -- 172.21.15.194:0/2316264980 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef008e1d0 con 0x7f7ef008c720 2024-04-26T09:35:40.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f7eeffff700 1 --2- 172.21.15.194:0/2316264980 >> v2:172.21.15.194:3300/0 conn(0x7f7ef0090df0 0x7f7ef008c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:40.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb0f24700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac12abf0 0x7f0eac12cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb0f24700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac12abf0 0x7f0eac12cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:46998/0 (socket says 172.21.15.194:46998) 2024-04-26T09:35:40.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb0f24700 1 -- 172.21.15.194:0/3621901876 learned_addr learned my addr 172.21.15.194:0/3621901876 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eabfff700 1 --1- 172.21.15.194:0/3621901876 >> v1:172.21.15.194:6789/0 conn(0x7f0eac08eb60 0x7f0eac0903c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:55190/0 (socket says 172.21.15.194:55190) 2024-04-26T09:35:40.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f7eef7fe700 1 --2- 172.21.15.194:0/2316264980 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008c720 0x7f7ef008cb60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f7ee4006fd0 tx=0x7f7ee40090f0).ready entity=mon.2 client_cookie=e5eb4141d8691f68 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb0f24700 1 -- 172.21.15.194:0/3621901876 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0eac090ad0 msgr2=0x7f0eac090eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb0f24700 1 --2- 172.21.15.194:0/3621901876 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0eac090ad0 0x7f0eac090eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:40.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb0f24700 1 -- 172.21.15.194:0/3621901876 >> v1:172.21.15.194:6789/0 conn(0x7f0eac08eb60 legacy=0x7f0eac0903c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb0f24700 1 -- 172.21.15.194:0/3621901876 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac0659a0 con 0x7f0eac12abf0 2024-04-26T09:35:40.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f0eb0f24700 1 --2- 172.21.15.194:0/3621901876 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac12abf0 0x7f0eac12cfe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f0ea4009f50 tx=0x7f0ea4009840).ready entity=mon.2 client_cookie=7414d2d035e9f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.483+0000 7f7eeeffd700 1 -- 172.21.15.194:0/2316264980 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee400b580 con 0x7f7ef008c720 2024-04-26T09:35:40.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f7eeeffd700 1 -- 172.21.15.194:0/2316264980 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee400b6e0 con 0x7f7ef008c720 2024-04-26T09:35:40.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f7eeeffd700 1 -- 172.21.15.194:0/2316264980 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee400ba60 con 0x7f7ef008c720 2024-04-26T09:35:40.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f0eaaffd700 1 -- 172.21.15.194:0/3621901876 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea400e070 con 0x7f0eac12abf0 2024-04-26T09:35:40.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f0eaaffd700 1 -- 172.21.15.194:0/3621901876 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea400bab0 con 0x7f0eac12abf0 2024-04-26T09:35:40.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f0eaaffd700 1 -- 172.21.15.194:0/3621901876 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea400be30 con 0x7f0eac12abf0 2024-04-26T09:35:40.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f7ef64c9700 1 -- 172.21.15.194:0/2316264980 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008c720 msgr2=0x7f7ef008cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f7ef64c9700 1 --2- 172.21.15.194:0/2316264980 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008c720 0x7f7ef008cb60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f7ee4006fd0 tx=0x7f7ee40090f0).stop 2024-04-26T09:35:40.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f0eb2987700 1 -- 172.21.15.194:0/3621901876 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac12abf0 msgr2=0x7f0eac12cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f0eb2987700 1 --2- 172.21.15.194:0/3621901876 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac12abf0 0x7f0eac12cfe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f0ea4009f50 tx=0x7f0ea4009840).stop 2024-04-26T09:35:40.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f7ef64c9700 1 -- 172.21.15.194:0/2316264980 shutdown_connections 2024-04-26T09:35:40.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f7ef64c9700 1 --2- 172.21.15.194:0/2316264980 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f7ef008b010 0x7f7ef008b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:40.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f7ef64c9700 1 --2- 172.21.15.194:0/2316264980 >> v2:172.21.15.194:3300/0 conn(0x7f7ef0090df0 0x7f7ef008c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f7ef64c9700 1 --2- 172.21.15.194:0/2316264980 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008c720 0x7f7ef008cb60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f7ef64c9700 1 -- 172.21.15.194:0/2316264980 >> 172.21.15.194:0/2316264980 conn(0x7f7ef0091b80 msgr2=0x7f7ef0091f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f7ef64c9700 1 -- 172.21.15.194:0/2316264980 shutdown_connections 2024-04-26T09:35:40.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f7ef64c9700 1 -- 172.21.15.194:0/2316264980 wait complete. 2024-04-26T09:35:40.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f0eb2987700 1 -- 172.21.15.194:0/3621901876 shutdown_connections 2024-04-26T09:35:40.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f0eb2987700 1 --2- 172.21.15.194:0/3621901876 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac12abf0 0x7f0eac12cfe0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f0eb2987700 1 --2- 172.21.15.194:0/3621901876 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0eac090ad0 0x7f0eac090eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:40.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.484+0000 7f0eb2987700 1 -- 172.21.15.194:0/3621901876 >> 172.21.15.194:0/3621901876 conn(0x7f0eac091f50 msgr2=0x7f0eac0949d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.485+0000 7f0eb2987700 1 -- 172.21.15.194:0/3621901876 shutdown_connections 2024-04-26T09:35:40.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.485+0000 7f0eb2987700 1 -- 172.21.15.194:0/3621901876 wait complete. 2024-04-26T09:35:40.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.485+0000 7f7ef64c9700 1 Processor -- start 2024-04-26T09:35:40.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.485+0000 7f0eb2987700 1 Processor -- start 2024-04-26T09:35:40.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f0eb2987700 1 -- start start 2024-04-26T09:35:40.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f7ef64c9700 1 -- start start 2024-04-26T09:35:40.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f0eb2987700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0eac08eb60 0x7f0eac08cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f0eb2987700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac090ad0 0x7f0eac08cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f0eb2987700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0eac12abf0 0x7f0eac08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f0eb2987700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac11b950 con 0x7f0eac12abf0 2024-04-26T09:35:40.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f0eb2987700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac08de40 con 0x7f0eac08eb60 2024-04-26T09:35:40.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f0eb2987700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac11b580 con 0x7f0eac090ad0 2024-04-26T09:35:40.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f0eab7fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac090ad0 0x7f0eac08cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f0eab7fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac090ad0 0x7f0eac08cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47002/0 (socket says 172.21.15.194:47002) 2024-04-26T09:35:40.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f0eab7fe700 1 -- 172.21.15.194:0/4094202103 learned_addr learned my addr 172.21.15.194:0/4094202103 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f0eb0f24700 1 --2- 172.21.15.194:0/4094202103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0eac12abf0 0x7f0eac08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f0eab7fe700 1 -- 172.21.15.194:0/4094202103 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0eac08eb60 msgr2=0x7f0eac08cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f0eab7fe700 1 --2- 172.21.15.194:0/4094202103 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0eac08eb60 0x7f0eac08cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:40.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f0eab7fe700 1 -- 172.21.15.194:0/4094202103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0eac12abf0 msgr2=0x7f0eac08d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f0eab7fe700 1 --2- 172.21.15.194:0/4094202103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0eac12abf0 0x7f0eac08d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f0eab7fe700 1 -- 172.21.15.194:0/4094202103 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea4009ae0 con 0x7f0eac090ad0 2024-04-26T09:35:40.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f7ef64c9700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008b010 0x7f7ef008a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f7ef64c9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ef008c720 0x7f7ef0130b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f7ef64c9700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f7ef0090df0 0x7f7ef0131080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f7ef64c9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef0088d30 con 0x7f7ef008c720 2024-04-26T09:35:40.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f7ef64c9700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef0119d50 con 0x7f7ef0090df0 2024-04-26T09:35:40.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.486+0000 7f7ef64c9700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef0119ec0 con 0x7f7ef008b010 2024-04-26T09:35:40.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f7eeffff700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008b010 0x7f7ef008a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f7eeffff700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008b010 0x7f7ef008a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47006/0 (socket says 172.21.15.194:47006) 2024-04-26T09:35:40.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f7eeffff700 1 -- 172.21.15.194:0/4226978534 learned_addr learned my addr 172.21.15.194:0/4226978534 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f7ef4a66700 1 --2- 172.21.15.194:0/4226978534 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f7ef0090df0 0x7f7ef0131080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f7eef7fe700 1 --2- 172.21.15.194:0/4226978534 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ef008c720 0x7f7ef0130b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f7eeffff700 1 -- 172.21.15.194:0/4226978534 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f7ef0090df0 msgr2=0x7f7ef0131080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f7eeffff700 1 --2- 172.21.15.194:0/4226978534 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f7ef0090df0 0x7f7ef0131080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f7eeffff700 1 -- 172.21.15.194:0/4226978534 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ef008c720 msgr2=0x7f7ef0130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f7eeffff700 1 --2- 172.21.15.194:0/4226978534 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ef008c720 0x7f7ef0130b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f7eeffff700 1 -- 172.21.15.194:0/4226978534 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee000c2d0 con 0x7f7ef008b010 2024-04-26T09:35:40.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f0eab7fe700 1 --2- 172.21.15.194:0/4094202103 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac090ad0 0x7f0eac08cfd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f0e9c00b530 tx=0x7f0e9c0090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f7eeffff700 1 --2- 172.21.15.194:0/4226978534 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008b010 0x7f7ef008a830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f7ee000fa80 tx=0x7f7ee0008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f0ea8ff9700 1 -- 172.21.15.194:0/4094202103 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c002320 con 0x7f0eac090ad0 2024-04-26T09:35:40.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f0ea8ff9700 1 -- 172.21.15.194:0/4094202103 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9c0024a0 con 0x7f0eac090ad0 2024-04-26T09:35:40.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f0ea8ff9700 1 -- 172.21.15.194:0/4094202103 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c002860 con 0x7f0eac090ad0 2024-04-26T09:35:40.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f0eb2987700 1 -- 172.21.15.194:0/4094202103 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac11aea0 con 0x7f0eac090ad0 2024-04-26T09:35:40.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.487+0000 7f0eb2987700 1 -- 172.21.15.194:0/4094202103 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac08da30 con 0x7f0eac090ad0 2024-04-26T09:35:40.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.488+0000 7f7eecff9700 1 -- 172.21.15.194:0/4226978534 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee000c550 con 0x7f7ef008b010 2024-04-26T09:35:40.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.488+0000 7f0eb2987700 1 -- 172.21.15.194:0/4094202103 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0eac0950e0 con 0x7f0eac090ad0 2024-04-26T09:35:40.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.488+0000 7f7eecff9700 1 -- 172.21.15.194:0/4226978534 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee000d960 con 0x7f7ef008b010 2024-04-26T09:35:40.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.488+0000 7f7ef64c9700 1 -- 172.21.15.194:0/4226978534 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee4009370 con 0x7f7ef008b010 2024-04-26T09:35:40.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.488+0000 7f7eecff9700 1 -- 172.21.15.194:0/4226978534 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee000dce0 con 0x7f7ef008b010 2024-04-26T09:35:40.507 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.488+0000 7f7ef64c9700 1 -- 172.21.15.194:0/4226978534 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef008df50 con 0x7f7ef008b010 2024-04-26T09:35:40.507 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.489+0000 7f0ea8ff9700 1 -- 172.21.15.194:0/4094202103 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f0e9c002c10 con 0x7f0eac090ad0 2024-04-26T09:35:40.507 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.489+0000 7f0ea8ff9700 1 -- 172.21.15.194:0/4094202103 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f0e9c0550b0 con 0x7f0eac090ad0 2024-04-26T09:35:40.507 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.489+0000 7f7ef64c9700 1 -- 172.21.15.194:0/4226978534 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7ef008c540 con 0x7f7ef008b010 2024-04-26T09:35:40.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.489+0000 7f0ea8ff9700 1 -- 172.21.15.194:0/4094202103 --> v1:172.21.15.194:6804/72866 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e9804bfa0 con 0x7f0e980482b0 2024-04-26T09:35:40.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.489+0000 7f0ea8ff9700 1 -- 172.21.15.194:0/4094202103 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e9c035bc0 con 0x7f0eac090ad0 2024-04-26T09:35:40.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.489+0000 7f7eecff9700 1 -- 172.21.15.194:0/4226978534 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f7ee0005a90 con 0x7f7ef008b010 2024-04-26T09:35:40.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.490+0000 7f7eecff9700 1 -- 172.21.15.194:0/4226978534 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f7ee0055460 con 0x7f7ef008b010 2024-04-26T09:35:40.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.490+0000 7f7eecff9700 1 -- 172.21.15.194:0/4226978534 --> v1:172.21.15.194:6800/72864 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7edc04c070 con 0x7f7edc048380 2024-04-26T09:35:40.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.490+0000 7f7eecff9700 1 -- 172.21.15.194:0/4226978534 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ee0057a70 con 0x7f7ef008b010 2024-04-26T09:35:40.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.490+0000 7f0ea8ff9700 1 -- 172.21.15.194:0/4094202103 <== osd.3 v1:172.21.15.194:6804/72866 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0e9804bfa0 con 0x7f0e980482b0 2024-04-26T09:35:40.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.491+0000 7f7eecff9700 1 -- 172.21.15.194:0/4226978534 <== osd.1 v1:172.21.15.194:6800/72864 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f7edc04c070 con 0x7f7edc048380 2024-04-26T09:35:40.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.504+0000 7f0eb2987700 1 -- 172.21.15.194:0/4094202103 --> v1:172.21.15.194:6804/72866 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0eac066080 con 0x7f0e980482b0 2024-04-26T09:35:40.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.505+0000 7f0ea8ff9700 1 -- 172.21.15.194:0/4094202103 <== osd.3 v1:172.21.15.194:6804/72866 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2217796425) 0x7f0eac066080 con 0x7f0e980482b0 2024-04-26T09:35:40.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.505+0000 7f0eb2987700 1 -- 172.21.15.194:0/4094202103 >> v1:172.21.15.194:6804/72866 conn(0x7f0e980482b0 legacy=0x7f0e9804a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.505+0000 7f0eb2987700 1 -- 172.21.15.194:0/4094202103 >> v1:172.21.15.194:6816/72244 conn(0x7f0e98041b80 legacy=0x7f0e98044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.505+0000 7f0eb2987700 1 -- 172.21.15.194:0/4094202103 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac090ad0 msgr2=0x7f0eac08cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.505+0000 7f0eb2987700 1 --2- 172.21.15.194:0/4094202103 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0eac090ad0 0x7f0eac08cfd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f0e9c00b530 tx=0x7f0e9c0090f0).stop 2024-04-26T09:35:40.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.505+0000 7f0eb0f24700 1 -- 172.21.15.194:0/4094202103 reap_dead start 2024-04-26T09:35:40.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.506+0000 7f0eb2987700 1 -- 172.21.15.194:0/4094202103 shutdown_connections 2024-04-26T09:35:40.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.506+0000 7f0eb2987700 1 -- 172.21.15.194:0/4094202103 >> 172.21.15.194:0/4094202103 conn(0x7f0eac091f50 msgr2=0x7f0eac069290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.506+0000 7f7ef64c9700 1 -- 172.21.15.194:0/4226978534 --> v1:172.21.15.194:6800/72864 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7ef0131a50 con 0x7f7edc048380 2024-04-26T09:35:40.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.506+0000 7f7eecff9700 1 -- 172.21.15.194:0/4226978534 <== osd.1 v1:172.21.15.194:6800/72864 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1985699914) 0x7f7ef0131a50 con 0x7f7edc048380 2024-04-26T09:35:40.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.506+0000 7f7ef64c9700 1 -- 172.21.15.194:0/4226978534 >> v1:172.21.15.194:6800/72864 conn(0x7f7edc048380 legacy=0x7f7edc04a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.506+0000 7f7ef64c9700 1 -- 172.21.15.194:0/4226978534 >> v1:172.21.15.194:6816/72244 conn(0x7f7edc041bd0 legacy=0x7f7edc044090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.506+0000 7f7ef64c9700 1 -- 172.21.15.194:0/4226978534 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008b010 msgr2=0x7f7ef008a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.506+0000 7f7ef64c9700 1 --2- 172.21.15.194:0/4226978534 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f7ef008b010 0x7f7ef008a830 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f7ee000fa80 tx=0x7f7ee0008040).stop 2024-04-26T09:35:40.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.506+0000 7f7ef4a66700 1 -- 172.21.15.194:0/4226978534 reap_dead start 2024-04-26T09:35:40.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.506+0000 7f7ef64c9700 1 -- 172.21.15.194:0/4226978534 shutdown_connections 2024-04-26T09:35:40.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.506+0000 7f7ef64c9700 1 -- 172.21.15.194:0/4226978534 >> 172.21.15.194:0/4226978534 conn(0x7f7ef0091b80 msgr2=0x7f7ef00679c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.507+0000 7f0eb2987700 1 -- 172.21.15.194:0/4094202103 shutdown_connections 2024-04-26T09:35:40.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.507+0000 7f0eb2987700 1 -- 172.21.15.194:0/4094202103 wait complete. 2024-04-26T09:35:40.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.510+0000 7f7ef64c9700 1 -- 172.21.15.194:0/4226978534 shutdown_connections 2024-04-26T09:35:40.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.510+0000 7f7ef64c9700 1 -- 172.21.15.194:0/4226978534 wait complete. 2024-04-26T09:35:40.523 INFO:teuthology.orchestra.run.smithi194.stdout:38654705667 2024-04-26T09:35:40.524 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T09:35:40.528 INFO:teuthology.orchestra.run.smithi194.stdout:38654705666 2024-04-26T09:35:40.529 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T09:35:40.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.525+0000 7f437a3f0700 1 Processor -- start 2024-04-26T09:35:40.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.525+0000 7f437a3f0700 1 -- start start 2024-04-26T09:35:40.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.525+0000 7f437a3f0700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f4374065de0 0x7f43740661c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.525+0000 7f437a3f0700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f43740666b0 0x7f4374121f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.525+0000 7f437a3f0700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4374069840 con 0x7f4374122410 2024-04-26T09:35:40.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.525+0000 7f437a3f0700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4374066f00 con 0x7f4374065de0 2024-04-26T09:35:40.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.525+0000 7f437a3f0700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f43740691e0 con 0x7f43740666b0 2024-04-26T09:35:40.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f43737fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f43740666b0 0x7f4374121f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f4373fff700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f4374065de0 0x7f43740661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f43737fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f43740666b0 0x7f4374121f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47022/0 (socket says 172.21.15.194:47022) 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f43737fe700 1 -- 172.21.15.194:0/837448651 learned_addr learned my addr 172.21.15.194:0/837448651 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f43737fe700 1 -- 172.21.15.194:0/837448651 >> v2:172.21.15.194:3300/0 conn(0x7f4374065de0 msgr2=0x7f43740661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f43737fe700 1 --2- 172.21.15.194:0/837448651 >> v2:172.21.15.194:3300/0 conn(0x7f4374065de0 0x7f43740661c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f43737fe700 1 -- 172.21.15.194:0/837448651 >> v1:172.21.15.194:6789/0 conn(0x7f4374122410 legacy=0x7f43741228a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f43737fe700 1 -- 172.21.15.194:0/837448651 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43740694f0 con 0x7f43740666b0 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f4373fff700 1 --2- 172.21.15.194:0/837448651 >> v2:172.21.15.194:3300/0 conn(0x7f4374065de0 0x7f43740661c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f43737fe700 1 --2- 172.21.15.194:0/837448651 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f43740666b0 0x7f4374121f20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4368006fd0 tx=0x7f43680090f0).ready entity=mon.2 client_cookie=c8d9388fb6bf267d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f4372ffd700 1 -- 172.21.15.194:0/837448651 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436800f510 con 0x7f43740666b0 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f4372ffd700 1 -- 172.21.15.194:0/837448651 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436800f670 con 0x7f43740666b0 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.526+0000 7f4372ffd700 1 -- 172.21.15.194:0/837448651 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436800fa10 con 0x7f43740666b0 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.527+0000 7f437a3f0700 1 -- 172.21.15.194:0/837448651 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f43740666b0 msgr2=0x7f4374121f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.527+0000 7f437a3f0700 1 --2- 172.21.15.194:0/837448651 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f43740666b0 0x7f4374121f20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f4368006fd0 tx=0x7f43680090f0).stop 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.527+0000 7f437a3f0700 1 -- 172.21.15.194:0/837448651 shutdown_connections 2024-04-26T09:35:40.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.527+0000 7f437a3f0700 1 --2- 172.21.15.194:0/837448651 >> v2:172.21.15.194:3300/0 conn(0x7f4374065de0 0x7f43740661c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.527+0000 7f437a3f0700 1 --2- 172.21.15.194:0/837448651 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f43740666b0 0x7f4374121f20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.527+0000 7f437a3f0700 1 -- 172.21.15.194:0/837448651 >> 172.21.15.194:0/837448651 conn(0x7f4374095750 msgr2=0x7f4374095b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.528+0000 7f6439df5700 1 Processor -- start 2024-04-26T09:35:40.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.528+0000 7f6439df5700 1 -- start start 2024-04-26T09:35:40.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.527+0000 7f437a3f0700 1 -- 172.21.15.194:0/837448651 shutdown_connections 2024-04-26T09:35:40.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.527+0000 7f437a3f0700 1 -- 172.21.15.194:0/837448651 wait complete. 2024-04-26T09:35:40.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.528+0000 7f6439df5700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f643408bb00 0x7f6434127640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.528+0000 7f6439df5700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f6434127b30 0x7f6434129f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.528+0000 7f6439df5700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643406a940 con 0x7f643408b010 2024-04-26T09:35:40.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.528+0000 7f6439df5700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6434090ef0 con 0x7f6434127b30 2024-04-26T09:35:40.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.528+0000 7f6439df5700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64340926a0 con 0x7f643408bb00 2024-04-26T09:35:40.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.529+0000 7f437a3f0700 1 Processor -- start 2024-04-26T09:35:40.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.529+0000 7f6433fff700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f6434127b30 0x7f6434129f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.529+0000 7f6433fff700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f6434127b30 0x7f6434129f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55228/0 (socket says 172.21.15.194:55228) 2024-04-26T09:35:40.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.529+0000 7f6433fff700 1 -- 172.21.15.194:0/2187855046 learned_addr learned my addr 172.21.15.194:0/2187855046 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.529+0000 7f6432ffd700 1 --2- 172.21.15.194:0/2187855046 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f643408bb00 0x7f6434127640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.529+0000 7f6433fff700 1 -- 172.21.15.194:0/2187855046 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f643408bb00 msgr2=0x7f6434127640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.529+0000 7f6433fff700 1 --2- 172.21.15.194:0/2187855046 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f643408bb00 0x7f6434127640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.529+0000 7f6433fff700 1 -- 172.21.15.194:0/2187855046 >> v1:172.21.15.194:6789/0 conn(0x7f643408b010 legacy=0x7f643408b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.529+0000 7f6433fff700 1 -- 172.21.15.194:0/2187855046 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643408e460 con 0x7f6434127b30 2024-04-26T09:35:40.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.529+0000 7f6433fff700 1 --2- 172.21.15.194:0/2187855046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6434127b30 0x7f6434129f20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f642c009f50 tx=0x7f642c009840).ready entity=mon.0 client_cookie=d20e559f0df884ea server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.529+0000 7f64327fc700 1 -- 172.21.15.194:0/2187855046 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c00e070 con 0x7f6434127b30 2024-04-26T09:35:40.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f64327fc700 1 -- 172.21.15.194:0/2187855046 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642c00b980 con 0x7f6434127b30 2024-04-26T09:35:40.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f64327fc700 1 -- 172.21.15.194:0/2187855046 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c00bd00 con 0x7f6434127b30 2024-04-26T09:35:40.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f437a3f0700 1 -- start start 2024-04-26T09:35:40.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f437a3f0700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4374065de0 0x7f4374139300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f437a3f0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43740666b0 0x7f43741397f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f437a3f0700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4374122410 0x7f4374112e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f437a3f0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4374115db0 con 0x7f43740666b0 2024-04-26T09:35:40.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f437a3f0700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43741149b0 con 0x7f4374065de0 2024-04-26T09:35:40.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f437a3f0700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f43741217f0 con 0x7f4374122410 2024-04-26T09:35:40.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f6439df5700 1 -- 172.21.15.194:0/2187855046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6434127b30 msgr2=0x7f6434129f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f6439df5700 1 --2- 172.21.15.194:0/2187855046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6434127b30 0x7f6434129f20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f642c009f50 tx=0x7f642c009840).stop 2024-04-26T09:35:40.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f4373fff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4374065de0 0x7f4374139300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f4373fff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4374065de0 0x7f4374139300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:60346/0 (socket says 172.21.15.194:60346) 2024-04-26T09:35:40.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f4373fff700 1 -- 172.21.15.194:0/2062020318 learned_addr learned my addr 172.21.15.194:0/2062020318 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f6439df5700 1 -- 172.21.15.194:0/2187855046 shutdown_connections 2024-04-26T09:35:40.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f6439df5700 1 --2- 172.21.15.194:0/2187855046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6434127b30 0x7f6434129f20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f6439df5700 1 --2- 172.21.15.194:0/2187855046 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f643408bb00 0x7f6434127640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.539 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f6439df5700 1 -- 172.21.15.194:0/2187855046 >> 172.21.15.194:0/2187855046 conn(0x7f6434091b80 msgr2=0x7f6434091f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f6439df5700 1 -- 172.21.15.194:0/2187855046 shutdown_connections 2024-04-26T09:35:40.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.530+0000 7f43737fe700 1 --2- 172.21.15.194:0/2062020318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43740666b0 0x7f43741397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f4373fff700 1 -- 172.21.15.194:0/2062020318 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4374122410 msgr2=0x7f4374112e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T09:35:40.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f4373fff700 1 --2- 172.21.15.194:0/2062020318 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4374122410 0x7f4374112e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:40.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f4373fff700 1 -- 172.21.15.194:0/2062020318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43740666b0 msgr2=0x7f43741397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f4373fff700 1 --2- 172.21.15.194:0/2062020318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43740666b0 0x7f43741397f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f4373fff700 1 -- 172.21.15.194:0/2062020318 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43640092a0 con 0x7f4374065de0 2024-04-26T09:35:40.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f4373fff700 1 --2- 172.21.15.194:0/2062020318 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4374065de0 0x7f4374139300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f436400ea50 tx=0x7f4364009060).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f4370ff9700 1 -- 172.21.15.194:0/2062020318 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364009600 con 0x7f4374065de0 2024-04-26T09:35:40.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f6439df5700 1 -- 172.21.15.194:0/2187855046 wait complete. 2024-04-26T09:35:40.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f6439df5700 1 Processor -- start 2024-04-26T09:35:40.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f4370ff9700 1 -- 172.21.15.194:0/2062020318 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436400c850 con 0x7f4374065de0 2024-04-26T09:35:40.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f437a3f0700 1 -- 172.21.15.194:0/2062020318 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4368009370 con 0x7f4374065de0 2024-04-26T09:35:40.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f4370ff9700 1 -- 172.21.15.194:0/2062020318 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436400cbd0 con 0x7f4374065de0 2024-04-26T09:35:40.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.531+0000 7f437a3f0700 1 -- 172.21.15.194:0/2062020318 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4374139ce0 con 0x7f4374065de0 2024-04-26T09:35:40.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f437a3f0700 1 -- 172.21.15.194:0/2062020318 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4374139f90 con 0x7f4374065de0 2024-04-26T09:35:40.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f6439df5700 1 -- start start 2024-04-26T09:35:40.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f6439df5700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f643408b010 0x7f6434125ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f6439df5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f643408bb00 0x7f6434126090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f6439df5700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6434127b30 0x7f6434126580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f6439df5700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6434068230 con 0x7f643408bb00 2024-04-26T09:35:40.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f6439df5700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6434068dd0 con 0x7f6434127b30 2024-04-26T09:35:40.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f6439df5700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f64340694d0 con 0x7f643408b010 2024-04-26T09:35:40.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f64337fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f643408b010 0x7f6434125ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f64337fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f643408b010 0x7f6434125ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47026/0 (socket says 172.21.15.194:47026) 2024-04-26T09:35:40.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f64337fe700 1 -- 172.21.15.194:0/2399582563 learned_addr learned my addr 172.21.15.194:0/2399582563 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.532+0000 7f6432ffd700 1 --2- 172.21.15.194:0/2399582563 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f643408bb00 0x7f6434126090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f6433fff700 1 --2- 172.21.15.194:0/2399582563 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6434127b30 0x7f6434126580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f6432ffd700 1 -- 172.21.15.194:0/2399582563 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f643408b010 msgr2=0x7f6434125ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f6432ffd700 1 --2- 172.21.15.194:0/2399582563 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f643408b010 0x7f6434125ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f6432ffd700 1 -- 172.21.15.194:0/2399582563 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6434127b30 msgr2=0x7f6434126580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f6432ffd700 1 --2- 172.21.15.194:0/2399582563 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f6434127b30 0x7f6434126580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f6432ffd700 1 -- 172.21.15.194:0/2399582563 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c009ae0 con 0x7f643408bb00 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f6432ffd700 1 --2- 172.21.15.194:0/2399582563 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f643408bb00 0x7f6434126090 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f642800e8d0 tx=0x7f6428009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f641bfff700 1 -- 172.21.15.194:0/2399582563 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428009470 con 0x7f643408bb00 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f6439df5700 1 -- 172.21.15.194:0/2399582563 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64340699e0 con 0x7f643408bb00 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f641bfff700 1 -- 172.21.15.194:0/2399582563 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6428012600 con 0x7f643408bb00 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f641bfff700 1 -- 172.21.15.194:0/2399582563 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428012980 con 0x7f643408bb00 2024-04-26T09:35:40.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.533+0000 7f6439df5700 1 -- 172.21.15.194:0/2399582563 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6434126cb0 con 0x7f643408bb00 2024-04-26T09:35:40.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.534+0000 7f4370ff9700 1 -- 172.21.15.194:0/2062020318 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f436401b400 con 0x7f4374065de0 2024-04-26T09:35:40.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.534+0000 7f4370ff9700 1 -- 172.21.15.194:0/2062020318 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f4364055e20 con 0x7f4374065de0 2024-04-26T09:35:40.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.534+0000 7f4370ff9700 1 -- 172.21.15.194:0/2062020318 --> v1:172.21.15.194:6812/72901 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f436004c140 con 0x7f4360048450 2024-04-26T09:35:40.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.534+0000 7f4370ff9700 1 -- 172.21.15.194:0/2062020318 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4364056150 con 0x7f4374065de0 2024-04-26T09:35:40.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.534+0000 7f6439df5700 1 -- 172.21.15.194:0/2399582563 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f643408eee0 con 0x7f643408bb00 2024-04-26T09:35:40.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.535+0000 7f641bfff700 1 -- 172.21.15.194:0/2399582563 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f6428012d30 con 0x7f643408bb00 2024-04-26T09:35:40.546 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.535+0000 7f641bfff700 1 -- 172.21.15.194:0/2399582563 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f6428037fa0 con 0x7f643408bb00 2024-04-26T09:35:40.546 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.535+0000 7f4370ff9700 1 -- 172.21.15.194:0/2062020318 <== osd.2 v1:172.21.15.194:6812/72901 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f436004c140 con 0x7f4360048450 2024-04-26T09:35:40.546 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.535+0000 7f641bfff700 1 -- 172.21.15.194:0/2399582563 --> v1:172.21.15.194:6808/72865 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f642004bfa0 con 0x7f64200482b0 2024-04-26T09:35:40.546 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.535+0000 7f641bfff700 1 -- 172.21.15.194:0/2399582563 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64280361a0 con 0x7f643408bb00 2024-04-26T09:35:40.546 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.537+0000 7f641bfff700 1 -- 172.21.15.194:0/2399582563 <== osd.0 v1:172.21.15.194:6808/72865 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f642004bfa0 con 0x7f64200482b0 2024-04-26T09:35:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.564+0000 7f437a3f0700 1 -- 172.21.15.194:0/2062020318 --> v1:172.21.15.194:6812/72901 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4374092a10 con 0x7f4360048450 2024-04-26T09:35:40.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.564+0000 7f4370ff9700 1 -- 172.21.15.194:0/2062020318 <== osd.2 v1:172.21.15.194:6812/72901 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1985699914) 0x7f4374092a10 con 0x7f4360048450 2024-04-26T09:35:40.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.564+0000 7f437a3f0700 1 -- 172.21.15.194:0/2062020318 >> v1:172.21.15.194:6812/72901 conn(0x7f4360048450 legacy=0x7f436004a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.565+0000 7f437a3f0700 1 -- 172.21.15.194:0/2062020318 >> v1:172.21.15.194:6816/72244 conn(0x7f4360041bd0 legacy=0x7f4360044090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.565+0000 7f437a3f0700 1 -- 172.21.15.194:0/2062020318 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4374065de0 msgr2=0x7f4374139300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.565+0000 7f437a3f0700 1 --2- 172.21.15.194:0/2062020318 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4374065de0 0x7f4374139300 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f436400ea50 tx=0x7f4364009060).stop 2024-04-26T09:35:40.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.565+0000 7f437898d700 1 -- 172.21.15.194:0/2062020318 reap_dead start 2024-04-26T09:35:40.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.565+0000 7f437a3f0700 1 -- 172.21.15.194:0/2062020318 shutdown_connections 2024-04-26T09:35:40.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.565+0000 7f437a3f0700 1 -- 172.21.15.194:0/2062020318 >> 172.21.15.194:0/2062020318 conn(0x7f4374095750 msgr2=0x7f4374120ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.565+0000 7f437a3f0700 1 -- 172.21.15.194:0/2062020318 shutdown_connections 2024-04-26T09:35:40.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.565+0000 7f437a3f0700 1 -- 172.21.15.194:0/2062020318 wait complete. 2024-04-26T09:35:40.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.565+0000 7f6439df5700 1 -- 172.21.15.194:0/2399582563 --> v1:172.21.15.194:6808/72865 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f64340922a0 con 0x7f64200482b0 2024-04-26T09:35:40.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.566+0000 7f641bfff700 1 -- 172.21.15.194:0/2399582563 <== osd.0 v1:172.21.15.194:6808/72865 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2217796425) 0x7f64340922a0 con 0x7f64200482b0 2024-04-26T09:35:40.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.566+0000 7f6439df5700 1 -- 172.21.15.194:0/2399582563 >> v1:172.21.15.194:6808/72865 conn(0x7f64200482b0 legacy=0x7f642004a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.567+0000 7f6439df5700 1 -- 172.21.15.194:0/2399582563 >> v1:172.21.15.194:6816/72244 conn(0x7f6420041b80 legacy=0x7f6420044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.567+0000 7f6439df5700 1 -- 172.21.15.194:0/2399582563 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f643408bb00 msgr2=0x7f6434126090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.567+0000 7f6439df5700 1 --2- 172.21.15.194:0/2399582563 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f643408bb00 0x7f6434126090 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f642800e8d0 tx=0x7f6428009060).stop 2024-04-26T09:35:40.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.567+0000 7f6433fff700 1 -- 172.21.15.194:0/2399582563 reap_dead start 2024-04-26T09:35:40.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.567+0000 7f6439df5700 1 -- 172.21.15.194:0/2399582563 shutdown_connections 2024-04-26T09:35:40.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.567+0000 7f6439df5700 1 -- 172.21.15.194:0/2399582563 >> 172.21.15.194:0/2399582563 conn(0x7f6434091b80 msgr2=0x7f6434093940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.567+0000 7f6439df5700 1 -- 172.21.15.194:0/2399582563 shutdown_connections 2024-04-26T09:35:40.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.567+0000 7f6439df5700 1 -- 172.21.15.194:0/2399582563 wait complete. 2024-04-26T09:35:40.587 INFO:teuthology.orchestra.run.smithi194.stdout:38654705666 2024-04-26T09:35:40.588 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T09:35:40.589 INFO:teuthology.orchestra.run.smithi194.stdout:38654705667 2024-04-26T09:35:40.589 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T09:35:40.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.649+0000 7fdc5d658700 1 Processor -- start 2024-04-26T09:35:40.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.649+0000 7fdc5d658700 1 -- start start 2024-04-26T09:35:40.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc5d658700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fdc580664c0 0x7fdc580668a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc5d658700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fdc581259c0 0x7fdc58125e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc5d658700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc58069b40 con 0x7fdc58066d90 2024-04-26T09:35:40.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc5d658700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc58069550 con 0x7fdc581259c0 2024-04-26T09:35:40.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc5d658700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdc58068990 con 0x7fdc580664c0 2024-04-26T09:35:40.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc56ffd700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fdc580664c0 0x7fdc580668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.654 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7f513d352700 1 Processor -- start 2024-04-26T09:35:40.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc56ffd700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fdc580664c0 0x7fdc580668a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47030/0 (socket says 172.21.15.194:47030) 2024-04-26T09:35:40.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc56ffd700 1 -- 172.21.15.194:0/2239003273 learned_addr learned my addr 172.21.15.194:0/2239003273 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc577fe700 1 --2- 172.21.15.194:0/2239003273 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fdc581259c0 0x7fdc58125e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc56ffd700 1 -- 172.21.15.194:0/2239003273 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fdc581259c0 msgr2=0x7fdc58125e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc56ffd700 1 --2- 172.21.15.194:0/2239003273 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fdc581259c0 0x7fdc58125e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7fdc56ffd700 1 -- 172.21.15.194:0/2239003273 >> v1:172.21.15.194:6789/0 conn(0x7fdc58066d90 legacy=0x7fdc581252b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7fdc56ffd700 1 -- 172.21.15.194:0/2239003273 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc58068430 con 0x7fdc580664c0 2024-04-26T09:35:40.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7fdc56ffd700 1 --2- 172.21.15.194:0/2239003273 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fdc580664c0 0x7fdc580668a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fdc48009840 tx=0x7fdc48009060).ready entity=mon.2 client_cookie=851aa3871b63ae19 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7fdc55ffb700 1 -- 172.21.15.194:0/2239003273 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc48005880 con 0x7fdc580664c0 2024-04-26T09:35:40.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7fdc55ffb700 1 -- 172.21.15.194:0/2239003273 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc480059e0 con 0x7fdc580664c0 2024-04-26T09:35:40.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.650+0000 7f513d352700 1 -- start start 2024-04-26T09:35:40.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f513d352700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f5138065f90 0x7f5138126c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f513d352700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f51381270f0 0x7f51381294e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f513d352700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5138091230 con 0x7f51380654a0 2024-04-26T09:35:40.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f513d352700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51380669c0 con 0x7f51381270f0 2024-04-26T09:35:40.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f513d352700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51380677c0 con 0x7f5138065f90 2024-04-26T09:35:40.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f51367fc700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f5138065f90 0x7f5138126c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f51367fc700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f5138065f90 0x7f5138126c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:60370/0 (socket says 172.21.15.194:60370) 2024-04-26T09:35:40.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f51367fc700 1 -- 172.21.15.194:0/2566439312 learned_addr learned my addr 172.21.15.194:0/2566439312 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7fdc55ffb700 1 -- 172.21.15.194:0/2239003273 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc48005d60 con 0x7fdc580664c0 2024-04-26T09:35:40.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f51377fe700 1 --2- 172.21.15.194:0/2566439312 >> v2:172.21.15.194:3300/0 conn(0x7f51381270f0 0x7f51381294e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f51367fc700 1 -- 172.21.15.194:0/2566439312 >> v2:172.21.15.194:3300/0 conn(0x7f51381270f0 msgr2=0x7f51381294e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f51367fc700 1 --2- 172.21.15.194:0/2566439312 >> v2:172.21.15.194:3300/0 conn(0x7f51381270f0 0x7f51381294e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f51367fc700 1 -- 172.21.15.194:0/2566439312 >> v1:172.21.15.194:6789/0 conn(0x7f51380654a0 legacy=0x7f5138065880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.651+0000 7f51367fc700 1 -- 172.21.15.194:0/2566439312 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5138067500 con 0x7f5138065f90 2024-04-26T09:35:40.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7f51367fc700 1 --2- 172.21.15.194:0/2566439312 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f5138065f90 0x7f5138126c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f512c006fd0 tx=0x7f512c00a2b0).ready entity=mon.1 client_cookie=c61391cb45466158 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7f5135ffb700 1 -- 172.21.15.194:0/2566439312 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c00ae90 con 0x7f5138065f90 2024-04-26T09:35:40.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7f5135ffb700 1 -- 172.21.15.194:0/2566439312 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c00c930 con 0x7f5138065f90 2024-04-26T09:35:40.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7f5135ffb700 1 -- 172.21.15.194:0/2566439312 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c00ccb0 con 0x7f5138065f90 2024-04-26T09:35:40.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7fdc5d658700 1 -- 172.21.15.194:0/2239003273 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fdc580664c0 msgr2=0x7fdc580668a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7fdc5d658700 1 --2- 172.21.15.194:0/2239003273 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fdc580664c0 0x7fdc580668a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fdc48009840 tx=0x7fdc48009060).stop 2024-04-26T09:35:40.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7fdc5d658700 1 -- 172.21.15.194:0/2239003273 shutdown_connections 2024-04-26T09:35:40.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7fdc5d658700 1 --2- 172.21.15.194:0/2239003273 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fdc581259c0 0x7fdc58125e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7fdc5d658700 1 --2- 172.21.15.194:0/2239003273 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fdc580664c0 0x7fdc580668a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7fdc5d658700 1 -- 172.21.15.194:0/2239003273 >> 172.21.15.194:0/2239003273 conn(0x7fdc58098a40 msgr2=0x7fdc58098e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7fdc5d658700 1 -- 172.21.15.194:0/2239003273 shutdown_connections 2024-04-26T09:35:40.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.652+0000 7fdc5d658700 1 -- 172.21.15.194:0/2239003273 wait complete. 2024-04-26T09:35:40.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.653+0000 7f513d352700 1 -- 172.21.15.194:0/2566439312 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f5138065f90 msgr2=0x7f5138126c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.653+0000 7f513d352700 1 --2- 172.21.15.194:0/2566439312 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f5138065f90 0x7f5138126c00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f512c006fd0 tx=0x7f512c00a2b0).stop 2024-04-26T09:35:40.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.653+0000 7fdc5d658700 1 Processor -- start 2024-04-26T09:35:40.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.653+0000 7f513d352700 1 -- 172.21.15.194:0/2566439312 shutdown_connections 2024-04-26T09:35:40.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.653+0000 7f513d352700 1 --2- 172.21.15.194:0/2566439312 >> v2:172.21.15.194:3300/0 conn(0x7f51381270f0 0x7f51381294e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.653+0000 7f513d352700 1 --2- 172.21.15.194:0/2566439312 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f5138065f90 0x7f5138126c00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.653+0000 7f513d352700 1 -- 172.21.15.194:0/2566439312 >> 172.21.15.194:0/2566439312 conn(0x7f51380914c0 msgr2=0x7f51380918d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.653+0000 7fdc5d658700 1 -- start start 2024-04-26T09:35:40.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.654+0000 7f513d352700 1 -- 172.21.15.194:0/2566439312 shutdown_connections 2024-04-26T09:35:40.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.654+0000 7f513d352700 1 -- 172.21.15.194:0/2566439312 wait complete. 2024-04-26T09:35:40.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.654+0000 7fdc5d658700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fdc580664c0 0x7fdc58117f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.654+0000 7fdc5d658700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fdc58066d90 0x7fdc58118400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.654+0000 7fdc5d658700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc581259c0 0x7fdc5811e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.654+0000 7fdc5d658700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc58084da0 con 0x7fdc581259c0 2024-04-26T09:35:40.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.654+0000 7fdc5d658700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc58084ae0 con 0x7fdc58066d90 2024-04-26T09:35:40.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.654+0000 7fdc5d658700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5811fb50 con 0x7fdc580664c0 2024-04-26T09:35:40.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.654+0000 7fdc577fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc581259c0 0x7fdc5811e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.654+0000 7fdc577fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc581259c0 0x7fdc5811e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55256/0 (socket says 172.21.15.194:55256) 2024-04-26T09:35:40.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.654+0000 7fdc577fe700 1 -- 172.21.15.194:0/4203198897 learned_addr learned my addr 172.21.15.194:0/4203198897 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7f513d352700 1 Processor -- start 2024-04-26T09:35:40.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7fdc577fe700 1 -- 172.21.15.194:0/4203198897 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fdc580664c0 msgr2=0x7fdc58117f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7fdc577fe700 1 --2- 172.21.15.194:0/4203198897 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fdc580664c0 0x7fdc58117f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:40.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7fdc577fe700 1 -- 172.21.15.194:0/4203198897 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fdc58066d90 msgr2=0x7fdc58118400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7fdc577fe700 1 --2- 172.21.15.194:0/4203198897 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fdc58066d90 0x7fdc58118400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:40.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7fdc577fe700 1 -- 172.21.15.194:0/4203198897 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc48009300 con 0x7fdc581259c0 2024-04-26T09:35:40.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7fdc577fe700 1 --2- 172.21.15.194:0/4203198897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc581259c0 0x7fdc5811e100 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fdc50009fa0 tx=0x7fdc50009900).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7fdc3f7fe700 1 -- 172.21.15.194:0/4203198897 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5000ed20 con 0x7fdc581259c0 2024-04-26T09:35:40.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7fdc3f7fe700 1 -- 172.21.15.194:0/4203198897 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5000ee80 con 0x7fdc581259c0 2024-04-26T09:35:40.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7fdc3f7fe700 1 -- 172.21.15.194:0/4203198897 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc50014610 con 0x7fdc581259c0 2024-04-26T09:35:40.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7fdc5d658700 1 -- 172.21.15.194:0/4203198897 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc58084710 con 0x7fdc581259c0 2024-04-26T09:35:40.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.655+0000 7f513d352700 1 -- start start 2024-04-26T09:35:40.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f513d352700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51380654a0 0x7f513808e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f513d352700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5138065f90 0x7f5138084630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f513d352700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f51381270f0 0x7f5138084b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f513d352700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513808eec0 con 0x7f51380654a0 2024-04-26T09:35:40.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f513d352700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5138085990 con 0x7f51381270f0 2024-04-26T09:35:40.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f513d352700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f51380878f0 con 0x7f5138065f90 2024-04-26T09:35:40.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7fdc5d658700 1 -- 172.21.15.194:0/4203198897 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc58118970 con 0x7fdc581259c0 2024-04-26T09:35:40.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f5136ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51380654a0 0x7f513808e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f51367fc700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5138065f90 0x7f5138084630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f5136ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51380654a0 0x7f513808e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55264/0 (socket says 172.21.15.194:55264) 2024-04-26T09:35:40.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f5136ffd700 1 -- 172.21.15.194:0/2085085164 learned_addr learned my addr 172.21.15.194:0/2085085164 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f51377fe700 1 --2- 172.21.15.194:0/2085085164 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f51381270f0 0x7f5138084b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f51367fc700 1 -- 172.21.15.194:0/2085085164 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f51381270f0 msgr2=0x7f5138084b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f51367fc700 1 --2- 172.21.15.194:0/2085085164 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f51381270f0 0x7f5138084b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f51367fc700 1 -- 172.21.15.194:0/2085085164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51380654a0 msgr2=0x7f513808e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f51367fc700 1 --2- 172.21.15.194:0/2085085164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51380654a0 0x7f513808e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f51367fc700 1 -- 172.21.15.194:0/2085085164 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c00a510 con 0x7f5138065f90 2024-04-26T09:35:40.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.656+0000 7f51367fc700 1 --2- 172.21.15.194:0/2085085164 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5138065f90 0x7f5138084630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f512c015000 tx=0x7f512c00a2b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.657+0000 7f511f7fe700 1 -- 172.21.15.194:0/2085085164 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c006db0 con 0x7f5138065f90 2024-04-26T09:35:40.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.657+0000 7fdc3f7fe700 1 -- 172.21.15.194:0/4203198897 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fdc500149c0 con 0x7fdc581259c0 2024-04-26T09:35:40.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.657+0000 7f511f7fe700 1 -- 172.21.15.194:0/2085085164 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c010bc0 con 0x7f5138065f90 2024-04-26T09:35:40.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.657+0000 7f513d352700 1 -- 172.21.15.194:0/2085085164 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5138087cc0 con 0x7f5138065f90 2024-04-26T09:35:40.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.657+0000 7f511f7fe700 1 -- 172.21.15.194:0/2085085164 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512c019420 con 0x7f5138065f90 2024-04-26T09:35:40.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.657+0000 7f513d352700 1 -- 172.21.15.194:0/2085085164 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5138085090 con 0x7f5138065f90 2024-04-26T09:35:40.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.657+0000 7fdc3f7fe700 1 -- 172.21.15.194:0/4203198897 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fdc50037ae0 con 0x7fdc581259c0 2024-04-26T09:35:40.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.658+0000 7fdc5d658700 1 -- 172.21.15.194:0/4203198897 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc58118bc0 con 0x7fdc581259c0 2024-04-26T09:35:40.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.658+0000 7f513d352700 1 -- 172.21.15.194:0/2085085164 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5138091fe0 con 0x7f5138065f90 2024-04-26T09:35:40.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.659+0000 7f511f7fe700 1 -- 172.21.15.194:0/2085085164 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f512c019810 con 0x7f5138065f90 2024-04-26T09:35:40.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.659+0000 7f511f7fe700 1 -- 172.21.15.194:0/2085085164 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f512c067920 con 0x7f5138065f90 2024-04-26T09:35:40.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.662+0000 7fdc3f7fe700 1 -- 172.21.15.194:0/4203198897 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fdc58118bc0 con 0x7fdc581259c0 2024-04-26T09:35:40.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.662+0000 7f511f7fe700 1 -- 172.21.15.194:0/2085085164 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f512c035a70 con 0x7f5138065f90 2024-04-26T09:35:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde6de72700 1 Processor -- start 2024-04-26T09:35:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde6de72700 1 -- start start 2024-04-26T09:35:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde6de72700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fde68063900 0x7fde68066110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde6de72700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066600 0x7fde681252b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde6de72700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68066a40 0x7fde681257a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde6de72700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde68065cc0 con 0x7fde68063900 2024-04-26T09:35:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde6de72700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde68067390 con 0x7fde68066a40 2024-04-26T09:35:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde6de72700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fde68067550 con 0x7fde68066600 2024-04-26T09:35:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde677fe700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fde68063900 0x7fde68066110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde66ffd700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066600 0x7fde681252b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde677fe700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fde68063900 0x7fde68066110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55280/0 (socket says 172.21.15.194:55280) 2024-04-26T09:35:40.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.757+0000 7fde677fe700 1 -- 172.21.15.194:0/1844616417 learned_addr learned my addr 172.21.15.194:0/1844616417 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.758+0000 7fde67fff700 1 --2- 172.21.15.194:0/1844616417 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68066a40 0x7fde681257a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.758+0000 7fde66ffd700 1 -- 172.21.15.194:0/1844616417 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68066a40 msgr2=0x7fde681257a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.758+0000 7fde66ffd700 1 --2- 172.21.15.194:0/1844616417 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68066a40 0x7fde681257a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.758+0000 7fde66ffd700 1 -- 172.21.15.194:0/1844616417 >> v2:172.21.15.194:3300/0 conn(0x7fde68063900 msgr2=0x7fde68066110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.758+0000 7fde66ffd700 1 --2- 172.21.15.194:0/1844616417 >> v2:172.21.15.194:3300/0 conn(0x7fde68063900 0x7fde68066110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.758+0000 7fde66ffd700 1 -- 172.21.15.194:0/1844616417 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde68067870 con 0x7fde68066600 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.758+0000 7fde677fe700 1 --2- 172.21.15.194:0/1844616417 >> v2:172.21.15.194:3300/0 conn(0x7fde68063900 0x7fde68066110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.758+0000 7fde66ffd700 1 --2- 172.21.15.194:0/1844616417 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066600 0x7fde681252b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fde5c009760 tx=0x7fde5c009060).ready entity=mon.2 client_cookie=f7ea8a09ae72d554 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.758+0000 7fde667fc700 1 -- 172.21.15.194:0/1844616417 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c00d4a0 con 0x7fde68066600 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.758+0000 7fde667fc700 1 -- 172.21.15.194:0/1844616417 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5c00d600 con 0x7fde68066600 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.758+0000 7fde667fc700 1 -- 172.21.15.194:0/1844616417 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c00d980 con 0x7fde68066600 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.759+0000 7fde6de72700 1 -- 172.21.15.194:0/1844616417 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066600 msgr2=0x7fde681252b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.759+0000 7fde6de72700 1 --2- 172.21.15.194:0/1844616417 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066600 0x7fde681252b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fde5c009760 tx=0x7fde5c009060).stop 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.759+0000 7fde6de72700 1 -- 172.21.15.194:0/1844616417 shutdown_connections 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.759+0000 7fde6de72700 1 --2- 172.21.15.194:0/1844616417 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68066a40 0x7fde681257a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.759+0000 7fde6de72700 1 --2- 172.21.15.194:0/1844616417 >> v2:172.21.15.194:3300/0 conn(0x7fde68063900 0x7fde68066110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.759+0000 7fde6de72700 1 --2- 172.21.15.194:0/1844616417 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066600 0x7fde681252b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.759+0000 7fde6de72700 1 -- 172.21.15.194:0/1844616417 >> 172.21.15.194:0/1844616417 conn(0x7fde68095780 msgr2=0x7fde68095b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.759+0000 7fde6de72700 1 -- 172.21.15.194:0/1844616417 shutdown_connections 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.759+0000 7fde6de72700 1 -- 172.21.15.194:0/1844616417 wait complete. 2024-04-26T09:35:40.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.760+0000 7fde6de72700 1 Processor -- start 2024-04-26T09:35:40.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.760+0000 7fde6de72700 1 -- start start 2024-04-26T09:35:40.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.760+0000 7fde6de72700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68063900 0x7fde68117200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.760+0000 7fde6de72700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde68066600 0x7fde681176f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde6de72700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066a40 0x7fde68117be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde6de72700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde6811a8c0 con 0x7fde68066600 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde6de72700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde6811abd0 con 0x7fde68063900 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde6de72700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fde681229d0 con 0x7fde68066a40 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde677fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68063900 0x7fde68117200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde677fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68063900 0x7fde68117200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:60420/0 (socket says 172.21.15.194:60420) 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde677fe700 1 -- 172.21.15.194:0/1805718670 learned_addr learned my addr 172.21.15.194:0/1805718670 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde67fff700 1 --2- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066a40 0x7fde68117be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde66ffd700 1 --2- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde68066600 0x7fde681176f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde67fff700 1 -- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68063900 msgr2=0x7fde68117200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde67fff700 1 --2- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68063900 0x7fde68117200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde67fff700 1 -- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde68066600 msgr2=0x7fde681176f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde67fff700 1 --2- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde68066600 0x7fde681176f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde67fff700 1 -- 172.21.15.194:0/1805718670 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde58009370 con 0x7fde68066a40 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde677fe700 1 --2- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68063900 0x7fde68117200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:40.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde67fff700 1 --2- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066a40 0x7fde68117be0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fde60009e20 tx=0x7fde6000b040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.761+0000 7fde4ffff700 1 -- 172.21.15.194:0/1805718670 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6000caa0 con 0x7fde68066a40 2024-04-26T09:35:40.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.762+0000 7fde4ffff700 1 -- 172.21.15.194:0/1805718670 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6000f040 con 0x7fde68066a40 2024-04-26T09:35:40.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.762+0000 7fde6de72700 1 -- 172.21.15.194:0/1805718670 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5c0092a0 con 0x7fde68066a40 2024-04-26T09:35:40.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.762+0000 7fde4ffff700 1 -- 172.21.15.194:0/1805718670 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6000ce10 con 0x7fde68066a40 2024-04-26T09:35:40.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.762+0000 7fde6de72700 1 -- 172.21.15.194:0/1805718670 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68119fc0 con 0x7fde68066a40 2024-04-26T09:35:40.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.763+0000 7fde4ffff700 1 -- 172.21.15.194:0/1805718670 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fde60013610 con 0x7fde68066a40 2024-04-26T09:35:40.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.763+0000 7fde4ffff700 1 -- 172.21.15.194:0/1805718670 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fde600559a0 con 0x7fde68066a40 2024-04-26T09:35:40.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.763+0000 7fde6de72700 1 -- 172.21.15.194:0/1805718670 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde68122c50 con 0x7fde68066a40 2024-04-26T09:35:40.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.767+0000 7fde4ffff700 1 -- 172.21.15.194:0/1805718670 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fde60036510 con 0x7fde68066a40 2024-04-26T09:35:40.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.777+0000 7fee78a3a700 1 Processor -- start 2024-04-26T09:35:40.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.777+0000 7fee78a3a700 1 -- start start 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.777+0000 7fee78a3a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fee74065f90 0x7fee74126c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.777+0000 7fee78a3a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fee741270f0 0x7fee741294e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee78a3a700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee74091230 con 0x7fee740654a0 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee78a3a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee740669c0 con 0x7fee741270f0 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee78a3a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fee740677c0 con 0x7fee74065f90 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee72d9d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fee741270f0 0x7fee741294e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee7259c700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fee740654a0 0x7fee74065880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:55246/0 (socket says 172.21.15.194:55246) 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee7259c700 1 -- 172.21.15.194:0/3822950809 learned_addr learned my addr 172.21.15.194:0/3822950809 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee71d9b700 1 --2- 172.21.15.194:0/3822950809 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fee74065f90 0x7fee74126c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee72d9d700 1 -- 172.21.15.194:0/3822950809 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fee74065f90 msgr2=0x7fee74126c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee72d9d700 1 --2- 172.21.15.194:0/3822950809 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fee74065f90 0x7fee74126c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee72d9d700 1 -- 172.21.15.194:0/3822950809 >> v1:172.21.15.194:6789/0 conn(0x7fee740654a0 legacy=0x7fee74065880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee7159a700 1 -- 172.21.15.194:0/3822950809 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107346623 0 0) 0x7fee74091230 con 0x7fee740654a0 2024-04-26T09:35:40.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee72d9d700 1 -- 172.21.15.194:0/3822950809 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee74067500 con 0x7fee741270f0 2024-04-26T09:35:40.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.778+0000 7fee72d9d700 1 --2- 172.21.15.194:0/3822950809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fee741270f0 0x7fee741294e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fee6c009d80 tx=0x7fee6c009780).ready entity=mon.1 client_cookie=8af44c84adc845e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.779+0000 7fee7159a700 1 -- 172.21.15.194:0/3822950809 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c00cc70 con 0x7fee741270f0 2024-04-26T09:35:40.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.779+0000 7fee7159a700 1 -- 172.21.15.194:0/3822950809 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee6c00cdd0 con 0x7fee741270f0 2024-04-26T09:35:40.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.779+0000 7fee7159a700 1 -- 172.21.15.194:0/3822950809 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee6c011630 con 0x7fee741270f0 2024-04-26T09:35:40.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.779+0000 7fee78a3a700 1 -- 172.21.15.194:0/3822950809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fee741270f0 msgr2=0x7fee741294e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.779+0000 7fee78a3a700 1 --2- 172.21.15.194:0/3822950809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fee741270f0 0x7fee741294e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fee6c009d80 tx=0x7fee6c009780).stop 2024-04-26T09:35:40.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.779+0000 7fee78a3a700 1 -- 172.21.15.194:0/3822950809 shutdown_connections 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.779+0000 7fee78a3a700 1 --2- 172.21.15.194:0/3822950809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fee741270f0 0x7fee741294e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.779+0000 7fee78a3a700 1 --2- 172.21.15.194:0/3822950809 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fee74065f90 0x7fee74126c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.779+0000 7fee78a3a700 1 -- 172.21.15.194:0/3822950809 >> 172.21.15.194:0/3822950809 conn(0x7fee740914c0 msgr2=0x7fee740918d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.779+0000 7fee78a3a700 1 -- 172.21.15.194:0/3822950809 shutdown_connections 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.779+0000 7fee78a3a700 1 -- 172.21.15.194:0/3822950809 wait complete. 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.780+0000 7fee78a3a700 1 Processor -- start 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.780+0000 7fee78a3a700 1 -- start start 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee78a3a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee740654a0 0x7fee74122610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee78a3a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fee74065f90 0x7fee74122b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee78a3a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fee741270f0 0x7fee74122ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee78a3a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee74122120 con 0x7fee740654a0 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee78a3a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee741234e0 con 0x7fee741270f0 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee78a3a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fee7411fdd0 con 0x7fee74065f90 2024-04-26T09:35:40.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee7259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee740654a0 0x7fee74122610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee7259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee740654a0 0x7fee74122610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55306/0 (socket says 172.21.15.194:55306) 2024-04-26T09:35:40.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee7259c700 1 -- 172.21.15.194:0/3753343249 learned_addr learned my addr 172.21.15.194:0/3753343249 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:40.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee72d9d700 1 --2- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fee741270f0 0x7fee74122ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee71d9b700 1 --2- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fee74065f90 0x7fee74122b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee71d9b700 1 -- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fee741270f0 msgr2=0x7fee74122ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee71d9b700 1 --2- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fee741270f0 0x7fee74122ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee71d9b700 1 -- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee740654a0 msgr2=0x7fee74122610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee71d9b700 1 --2- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee740654a0 0x7fee74122610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee71d9b700 1 -- 172.21.15.194:0/3753343249 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c00d040 con 0x7fee74065f90 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.781+0000 7fee71d9b700 1 --2- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fee74065f90 0x7fee74122b00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fee6800e820 tx=0x7fee68009060).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.782+0000 7fee62ffd700 1 -- 172.21.15.194:0/3753343249 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee68009450 con 0x7fee74065f90 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.782+0000 7fee78a3a700 1 -- 172.21.15.194:0/3753343249 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee74120550 con 0x7fee74065f90 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.782+0000 7fee78a3a700 1 -- 172.21.15.194:0/3753343249 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee74138e50 con 0x7fee74065f90 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.782+0000 7fee62ffd700 1 -- 172.21.15.194:0/3753343249 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee68012600 con 0x7fee74065f90 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.782+0000 7fee62ffd700 1 -- 172.21.15.194:0/3753343249 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee680129a0 con 0x7fee74065f90 2024-04-26T09:35:40.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.783+0000 7fee62ffd700 1 -- 172.21.15.194:0/3753343249 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fee68012d50 con 0x7fee74065f90 2024-04-26T09:35:40.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.783+0000 7fee62ffd700 1 -- 172.21.15.194:0/3753343249 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fee68055140 con 0x7fee74065f90 2024-04-26T09:35:40.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.783+0000 7fee78a3a700 1 -- 172.21.15.194:0/3753343249 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee58005120 con 0x7fee74065f90 2024-04-26T09:35:40.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.787+0000 7fee62ffd700 1 -- 172.21.15.194:0/3753343249 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fee68016080 con 0x7fee74065f90 2024-04-26T09:35:40.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.814+0000 7f513d352700 1 -- 172.21.15.194:0/2085085164 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5138067170 con 0x7f5138065f90 2024-04-26T09:35:40.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.814+0000 7f511f7fe700 1 -- 172.21.15.194:0/2085085164 <== mon.2 v2:172.21.15.194:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f512c014070 con 0x7f5138065f90 2024-04-26T09:35:40.818 INFO:teuthology.orchestra.run.smithi194.stdout:38654705665 2024-04-26T09:35:40.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.816+0000 7f513d352700 1 -- 172.21.15.194:0/2085085164 >> v1:172.21.15.194:6816/72244 conn(0x7f5124041bb0 legacy=0x7f5124044070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.816+0000 7f513d352700 1 -- 172.21.15.194:0/2085085164 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5138065f90 msgr2=0x7f5138084630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.816+0000 7f513d352700 1 --2- 172.21.15.194:0/2085085164 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5138065f90 0x7f5138084630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f512c015000 tx=0x7f512c00a2b0).stop 2024-04-26T09:35:40.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.816+0000 7f513d352700 1 -- 172.21.15.194:0/2085085164 shutdown_connections 2024-04-26T09:35:40.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.816+0000 7f513d352700 1 --2- 172.21.15.194:0/2085085164 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f51381270f0 0x7f5138084b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.816+0000 7f513d352700 1 --2- 172.21.15.194:0/2085085164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51380654a0 0x7f513808e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.816+0000 7f513d352700 1 --2- 172.21.15.194:0/2085085164 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5138065f90 0x7f5138084630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.816+0000 7f513d352700 1 -- 172.21.15.194:0/2085085164 >> 172.21.15.194:0/2085085164 conn(0x7f51380914c0 msgr2=0x7f5138126910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.816+0000 7f513d352700 1 -- 172.21.15.194:0/2085085164 shutdown_connections 2024-04-26T09:35:40.820 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.816+0000 7f513d352700 1 -- 172.21.15.194:0/2085085164 wait complete. 2024-04-26T09:35:40.832 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.1 2024-04-26T09:35:40.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.856+0000 7fdc5d658700 1 -- 172.21.15.194:0/4203198897 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fdc58083e90 con 0x7fdc581259c0 2024-04-26T09:35:40.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.856+0000 7fdc3f7fe700 1 -- 172.21.15.194:0/4203198897 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdc58083e90 con 0x7fdc581259c0 2024-04-26T09:35:40.860 INFO:teuthology.orchestra.run.smithi194.stdout:38654705666 2024-04-26T09:35:40.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.858+0000 7fdc5d658700 1 -- 172.21.15.194:0/4203198897 >> v1:172.21.15.194:6816/72244 conn(0x7fdc44041b80 legacy=0x7fdc44044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.858+0000 7fdc5d658700 1 -- 172.21.15.194:0/4203198897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc581259c0 msgr2=0x7fdc5811e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.858+0000 7fdc5d658700 1 --2- 172.21.15.194:0/4203198897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc581259c0 0x7fdc5811e100 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fdc50009fa0 tx=0x7fdc50009900).stop 2024-04-26T09:35:40.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.858+0000 7fdc5d658700 1 -- 172.21.15.194:0/4203198897 shutdown_connections 2024-04-26T09:35:40.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.858+0000 7fdc5d658700 1 --2- 172.21.15.194:0/4203198897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc581259c0 0x7fdc5811e100 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.858+0000 7fdc5d658700 1 --2- 172.21.15.194:0/4203198897 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fdc580664c0 0x7fdc58117f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:40.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.858+0000 7fdc5d658700 1 --2- 172.21.15.194:0/4203198897 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fdc58066d90 0x7fdc58118400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:40.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.858+0000 7fdc5d658700 1 -- 172.21.15.194:0/4203198897 >> 172.21.15.194:0/4203198897 conn(0x7fdc58098a40 msgr2=0x7fdc580972d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.858+0000 7fdc5d658700 1 -- 172.21.15.194:0/4203198897 shutdown_connections 2024-04-26T09:35:40.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.858+0000 7fdc5d658700 1 -- 172.21.15.194:0/4203198897 wait complete. 2024-04-26T09:35:40.874 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.3 2024-04-26T09:35:40.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.933+0000 7fde6de72700 1 -- 172.21.15.194:0/1805718670 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fde68125c90 con 0x7fde68066a40 2024-04-26T09:35:40.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.933+0000 7fde4ffff700 1 -- 172.21.15.194:0/1805718670 <== mon.2 v2:172.21.15.194:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fde60039b30 con 0x7fde68066a40 2024-04-26T09:35:40.937 INFO:teuthology.orchestra.run.smithi194.stdout:38654705665 2024-04-26T09:35:40.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.935+0000 7fde6de72700 1 -- 172.21.15.194:0/1805718670 >> v1:172.21.15.194:6816/72244 conn(0x7fde54041b80 legacy=0x7fde54044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.935+0000 7fde6de72700 1 -- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066a40 msgr2=0x7fde68117be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.935+0000 7fde6de72700 1 --2- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066a40 0x7fde68117be0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fde60009e20 tx=0x7fde6000b040).stop 2024-04-26T09:35:40.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.935+0000 7fde6de72700 1 -- 172.21.15.194:0/1805718670 shutdown_connections 2024-04-26T09:35:40.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.935+0000 7fde6de72700 1 --2- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde68066a40 0x7fde68117be0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.935+0000 7fde6de72700 1 --2- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde68063900 0x7fde68117200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.935+0000 7fde6de72700 1 --2- 172.21.15.194:0/1805718670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde68066600 0x7fde681176f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.935+0000 7fde6de72700 1 -- 172.21.15.194:0/1805718670 >> 172.21.15.194:0/1805718670 conn(0x7fde68095780 msgr2=0x7fde68124780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.935+0000 7fde6de72700 1 -- 172.21.15.194:0/1805718670 shutdown_connections 2024-04-26T09:35:40.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.935+0000 7fde6de72700 1 -- 172.21.15.194:0/1805718670 wait complete. 2024-04-26T09:35:40.951 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.2 2024-04-26T09:35:40.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.950+0000 7fee78a3a700 1 -- 172.21.15.194:0/3753343249 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fee580057f0 con 0x7fee74065f90 2024-04-26T09:35:40.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.950+0000 7fee62ffd700 1 -- 172.21.15.194:0/3753343249 <== mon.2 v2:172.21.15.194:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fee68035a80 con 0x7fee74065f90 2024-04-26T09:35:40.954 INFO:teuthology.orchestra.run.smithi194.stdout:38654705666 2024-04-26T09:35:40.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.952+0000 7fee78a3a700 1 -- 172.21.15.194:0/3753343249 >> v1:172.21.15.194:6816/72244 conn(0x7fee54041bb0 legacy=0x7fee54044070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.952+0000 7fee78a3a700 1 -- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fee74065f90 msgr2=0x7fee74122b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:40.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.952+0000 7fee78a3a700 1 --2- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fee74065f90 0x7fee74122b00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fee6800e820 tx=0x7fee68009060).stop 2024-04-26T09:35:40.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.952+0000 7fee78a3a700 1 -- 172.21.15.194:0/3753343249 shutdown_connections 2024-04-26T09:35:40.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.952+0000 7fee78a3a700 1 --2- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fee741270f0 0x7fee74122ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.952+0000 7fee78a3a700 1 --2- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee740654a0 0x7fee74122610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.952+0000 7fee78a3a700 1 --2- 172.21.15.194:0/3753343249 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fee74065f90 0x7fee74122b00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:40.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.952+0000 7fee78a3a700 1 -- 172.21.15.194:0/3753343249 >> 172.21.15.194:0/3753343249 conn(0x7fee740914c0 msgr2=0x7fee74125eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:40.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.952+0000 7fee78a3a700 1 -- 172.21.15.194:0/3753343249 shutdown_connections 2024-04-26T09:35:40.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:40.952+0000 7fee78a3a700 1 -- 172.21.15.194:0/3753343249 wait complete. 2024-04-26T09:35:40.967 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.0 2024-04-26T09:35:41.833 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T09:35:41.875 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T09:35:41.952 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T09:35:41.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.962+0000 7f2c0bce7700 1 Processor -- start 2024-04-26T09:35:41.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.962+0000 7f2c0bce7700 1 -- start start 2024-04-26T09:35:41.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0bce7700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04063900 0x7f2c0411fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:41.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0bce7700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f2c04069ab0 0x7f2c04120260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:41.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0bce7700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04120750 0x7f2c04123020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:41.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0bce7700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0411fb40 con 0x7f2c04069ab0 2024-04-26T09:35:41.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0bce7700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0410bc20 con 0x7f2c04063900 2024-04-26T09:35:41.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0bce7700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0410bd90 con 0x7f2c04120750 2024-04-26T09:35:41.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0a284700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04120750 0x7f2c04123020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:41.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0a284700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04120750 0x7f2c04123020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:32992/0 (socket says 172.21.15.194:32992) 2024-04-26T09:35:41.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c09282700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f2c04069ab0 0x7f2c04120260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:41.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0a284700 1 -- 172.21.15.194:0/1494921482 learned_addr learned my addr 172.21.15.194:0/1494921482 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:41.969 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c09a83700 1 --2- 172.21.15.194:0/1494921482 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04063900 0x7f2c0411fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0a284700 1 -- 172.21.15.194:0/1494921482 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04063900 msgr2=0x7f2c0411fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0a284700 1 --2- 172.21.15.194:0/1494921482 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04063900 0x7f2c0411fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0a284700 1 -- 172.21.15.194:0/1494921482 >> v2:172.21.15.194:3300/0 conn(0x7f2c04069ab0 msgr2=0x7f2c04120260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0a284700 1 --2- 172.21.15.194:0/1494921482 >> v2:172.21.15.194:3300/0 conn(0x7f2c04069ab0 0x7f2c04120260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0a284700 1 -- 172.21.15.194:0/1494921482 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c04123510 con 0x7f2c04120750 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c09a83700 1 --2- 172.21.15.194:0/1494921482 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04063900 0x7f2c0411fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c09282700 1 --2- 172.21.15.194:0/1494921482 >> v2:172.21.15.194:3300/0 conn(0x7f2c04069ab0 0x7f2c04120260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.963+0000 7f2c0a284700 1 --2- 172.21.15.194:0/1494921482 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04120750 0x7f2c04123020 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2bfc009f50 tx=0x7f2bfc009840).ready entity=mon.2 client_cookie=fb4b4e8eacfbb63d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.964+0000 7f2c08a81700 1 -- 172.21.15.194:0/1494921482 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc00e070 con 0x7f2c04120750 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.964+0000 7f2c08a81700 1 -- 172.21.15.194:0/1494921482 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bfc00b910 con 0x7f2c04120750 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.964+0000 7f2c08a81700 1 -- 172.21.15.194:0/1494921482 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc00bc90 con 0x7f2c04120750 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.964+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1494921482 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04120750 msgr2=0x7f2c04123020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.964+0000 7f2c0bce7700 1 --2- 172.21.15.194:0/1494921482 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04120750 0x7f2c04123020 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f2bfc009f50 tx=0x7f2bfc009840).stop 2024-04-26T09:35:41.970 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.964+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1494921482 shutdown_connections 2024-04-26T09:35:41.971 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.964+0000 7f2c0bce7700 1 --2- 172.21.15.194:0/1494921482 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04120750 0x7f2c04123020 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:41.971 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.964+0000 7f2c0bce7700 1 --2- 172.21.15.194:0/1494921482 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04063900 0x7f2c0411fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:41.971 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.964+0000 7f2c0bce7700 1 --2- 172.21.15.194:0/1494921482 >> v2:172.21.15.194:3300/0 conn(0x7f2c04069ab0 0x7f2c04120260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:41.971 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.964+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1494921482 >> 172.21.15.194:0/1494921482 conn(0x7f2c04095750 msgr2=0x7f2c04095b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:41.971 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.965+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1494921482 shutdown_connections 2024-04-26T09:35:41.971 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.965+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1494921482 wait complete. 2024-04-26T09:35:41.971 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.965+0000 7f2c0bce7700 1 Processor -- start 2024-04-26T09:35:41.971 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.965+0000 7f2c0bce7700 1 -- start start 2024-04-26T09:35:41.971 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T09:35:41.973 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c0bce7700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04063900 0x7f2c04118d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c0bce7700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04069ab0 0x7f2c041191f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c0bce7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c04120750 0x7f2c041196e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c0bce7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c04118810 con 0x7f2c04120750 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c0bce7700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c04119bd0 con 0x7f2c04069ab0 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c0bce7700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f2c04118b20 con 0x7f2c04063900 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c09a83700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04063900 0x7f2c04118d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c09a83700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04063900 0x7f2c04118d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:33000/0 (socket says 172.21.15.194:33000) 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c09a83700 1 -- 172.21.15.194:0/1040266975 learned_addr learned my addr 172.21.15.194:0/1040266975 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c09282700 1 --2- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04069ab0 0x7f2c041191f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c0a284700 1 --2- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c04120750 0x7f2c041196e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c09282700 1 -- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04063900 msgr2=0x7f2c04118d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c09282700 1 --2- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04063900 0x7f2c04118d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:41.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c09282700 1 -- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c04120750 msgr2=0x7f2c041196e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:41.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c09282700 1 --2- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c04120750 0x7f2c041196e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:41.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.966+0000 7f2c09282700 1 -- 172.21.15.194:0/1040266975 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf40092a0 con 0x7f2c04069ab0 2024-04-26T09:35:41.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.967+0000 7f2c09282700 1 --2- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04069ab0 0x7f2c041191f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f2c040640b0 tx=0x7f2bf8009060).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:41.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.967+0000 7f2c027fc700 1 -- 172.21.15.194:0/1040266975 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8009580 con 0x7f2c04069ab0 2024-04-26T09:35:41.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.967+0000 7f2c027fc700 1 -- 172.21.15.194:0/1040266975 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf8012600 con 0x7f2c04069ab0 2024-04-26T09:35:41.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.967+0000 7f2c027fc700 1 -- 172.21.15.194:0/1040266975 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf80129a0 con 0x7f2c04069ab0 2024-04-26T09:35:41.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.967+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1040266975 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf80092a0 con 0x7f2c04069ab0 2024-04-26T09:35:41.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.967+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1040266975 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bfc009ae0 con 0x7f2c04069ab0 2024-04-26T09:35:41.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.968+0000 7f2c027fc700 1 -- 172.21.15.194:0/1040266975 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f2bf8012d10 con 0x7f2c04069ab0 2024-04-26T09:35:41.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.968+0000 7f2c027fc700 1 -- 172.21.15.194:0/1040266975 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f2bf80554c0 con 0x7f2c04069ab0 2024-04-26T09:35:41.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.968+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1040266975 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c04092a10 con 0x7f2c04069ab0 2024-04-26T09:35:41.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:41.972+0000 7f2c027fc700 1 -- 172.21.15.194:0/1040266975 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f2bf8036090 con 0x7f2c04069ab0 2024-04-26T09:35:42.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.012+0000 7ffb6507c700 1 Processor -- start 2024-04-26T09:35:42.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.012+0000 7ffb6507c700 1 -- start start 2024-04-26T09:35:42.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.012+0000 7ffb6507c700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7ffb60066d30 0x7ffb60067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.012+0000 7ffb6507c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 0x7ffb6012b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.012+0000 7ffb6507c700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb60068740 con 0x7ffb60067600 2024-04-26T09:35:42.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.012+0000 7ffb6507c700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb60069650 con 0x7ffb60066d30 2024-04-26T09:35:42.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.012+0000 7ffb6507c700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ffb60068080 con 0x7ffb60120320 2024-04-26T09:35:42.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5ed9d700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7ffb60066d30 0x7ffb60067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5ed9d700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7ffb60066d30 0x7ffb60067110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48286/0 (socket says 172.21.15.194:48286) 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5ed9d700 1 -- 172.21.15.194:0/437507339 learned_addr learned my addr 172.21.15.194:0/437507339 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5f59e700 1 --2- 172.21.15.194:0/437507339 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 0x7ffb6012b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5ed9d700 1 -- 172.21.15.194:0/437507339 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 msgr2=0x7ffb6012b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5ed9d700 1 --2- 172.21.15.194:0/437507339 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 0x7ffb6012b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5ed9d700 1 -- 172.21.15.194:0/437507339 >> v1:172.21.15.194:6789/0 conn(0x7ffb60067600 legacy=0x7ffb6011fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5dd9b700 1 -- 172.21.15.194:0/437507339 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075473453 0 0) 0x7ffb60068740 con 0x7ffb60067600 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5ed9d700 1 -- 172.21.15.194:0/437507339 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb60069910 con 0x7ffb60066d30 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5f59e700 1 --2- 172.21.15.194:0/437507339 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 0x7ffb6012b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5ed9d700 1 --2- 172.21.15.194:0/437507339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb60066d30 0x7ffb60067110 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7ffb50006fd0 tx=0x7ffb500090f0).ready entity=mon.0 client_cookie=f38ab2195f3e53d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5dd9b700 1 -- 172.21.15.194:0/437507339 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5000b580 con 0x7ffb60066d30 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5dd9b700 1 -- 172.21.15.194:0/437507339 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5000b6e0 con 0x7ffb60066d30 2024-04-26T09:35:42.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.013+0000 7ffb5dd9b700 1 -- 172.21.15.194:0/437507339 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5000ba80 con 0x7ffb60066d30 2024-04-26T09:35:42.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.014+0000 7ffb6507c700 1 -- 172.21.15.194:0/437507339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb60066d30 msgr2=0x7ffb60067110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.014+0000 7ffb6507c700 1 --2- 172.21.15.194:0/437507339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb60066d30 0x7ffb60067110 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7ffb50006fd0 tx=0x7ffb500090f0).stop 2024-04-26T09:35:42.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.014+0000 7ffb6507c700 1 -- 172.21.15.194:0/437507339 shutdown_connections 2024-04-26T09:35:42.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.014+0000 7ffb6507c700 1 --2- 172.21.15.194:0/437507339 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 0x7ffb6012b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.014+0000 7ffb6507c700 1 --2- 172.21.15.194:0/437507339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb60066d30 0x7ffb60067110 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.014+0000 7ffb6507c700 1 -- 172.21.15.194:0/437507339 >> 172.21.15.194:0/437507339 conn(0x7ffb60096860 msgr2=0x7ffb60096c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:42.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.014+0000 7ffb6507c700 1 -- 172.21.15.194:0/437507339 shutdown_connections 2024-04-26T09:35:42.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.014+0000 7ffb6507c700 1 -- 172.21.15.194:0/437507339 wait complete. 2024-04-26T09:35:42.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.016+0000 7ffb6507c700 1 Processor -- start 2024-04-26T09:35:42.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.016+0000 7ffb6507c700 1 -- start start 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb6507c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb60066d30 0x7ffb60111970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb6507c700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ffb60067600 0x7ffb60111e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb6507c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 0x7ffb60112350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb6507c700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb601136a0 con 0x7ffb60066d30 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb6507c700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6011fb70 con 0x7ffb60067600 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb6507c700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ffb60112a20 con 0x7ffb60120320 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb5e59c700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ffb60067600 0x7ffb60111e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb5f59e700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 0x7ffb60112350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb5f59e700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 0x7ffb60112350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:33018/0 (socket says 172.21.15.194:33018) 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb5f59e700 1 -- 172.21.15.194:0/3332195121 learned_addr learned my addr 172.21.15.194:0/3332195121 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb5e59c700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ffb60067600 0x7ffb60111e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56804/0 (socket says 172.21.15.194:56804) 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.017+0000 7ffb5ed9d700 1 --2- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb60066d30 0x7ffb60111970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb5ed9d700 1 -- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 msgr2=0x7ffb60112350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.022 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb5ed9d700 1 --2- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 0x7ffb60112350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb5ed9d700 1 -- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ffb60067600 msgr2=0x7ffb60111e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb5ed9d700 1 --2- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ffb60067600 0x7ffb60111e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb5ed9d700 1 -- 172.21.15.194:0/3332195121 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb58009a20 con 0x7ffb60066d30 2024-04-26T09:35:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb5f59e700 1 --2- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 0x7ffb60112350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb5e59c700 1 --2- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ffb60067600 0x7ffb60111e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb5ed9d700 1 --2- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb60066d30 0x7ffb60111970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7ffb50014000 tx=0x7ffb50017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb477fe700 1 -- 172.21.15.194:0/3332195121 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb50006d90 con 0x7ffb60066d30 2024-04-26T09:35:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb477fe700 1 -- 172.21.15.194:0/3332195121 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5000be40 con 0x7ffb60066d30 2024-04-26T09:35:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb477fe700 1 -- 172.21.15.194:0/3332195121 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5001c420 con 0x7ffb60066d30 2024-04-26T09:35:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb6507c700 1 -- 172.21.15.194:0/3332195121 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb50009370 con 0x7ffb60066d30 2024-04-26T09:35:42.023 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.018+0000 7ffb6507c700 1 -- 172.21.15.194:0/3332195121 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb601147d0 con 0x7ffb60066d30 2024-04-26T09:35:42.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.019+0000 7ffb6507c700 1 -- 172.21.15.194:0/3332195121 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb60092a10 con 0x7ffb60066d30 2024-04-26T09:35:42.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.020+0000 7ffb477fe700 1 -- 172.21.15.194:0/3332195121 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7ffb5001c810 con 0x7ffb60066d30 2024-04-26T09:35:42.024 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.020+0000 7ffb477fe700 1 -- 172.21.15.194:0/3332195121 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7ffb500568a0 con 0x7ffb60066d30 2024-04-26T09:35:42.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.027+0000 7ffb477fe700 1 -- 172.21.15.194:0/3332195121 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7ffb500262e0 con 0x7ffb60066d30 2024-04-26T09:35:42.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.083+0000 7f73636ba700 1 Processor -- start 2024-04-26T09:35:42.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.083+0000 7f73636ba700 1 -- start start 2024-04-26T09:35:42.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.083+0000 7f73636ba700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 0x7f735c0903d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.083+0000 7f73636ba700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f735c090fd0 0x7f735c12cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.083+0000 7f73636ba700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735c05f9c0 con 0x7f735c05e230 2024-04-26T09:35:42.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.083+0000 7f73636ba700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735c066d70 con 0x7f735c090fd0 2024-04-26T09:35:42.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.083+0000 7f73636ba700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c05e8c0 con 0x7f735c08ecb0 2024-04-26T09:35:42.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.083+0000 7f7361c57700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f735c090fd0 0x7f735c12cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7360c55700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f735c05e230 0x7f735c0908c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:51694/0 (socket says 172.21.15.194:51694) 2024-04-26T09:35:42.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7360c55700 1 -- 172.21.15.194:0/1161101507 learned_addr learned my addr 172.21.15.194:0/1161101507 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:35:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7361456700 1 --2- 172.21.15.194:0/1161101507 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 0x7f735c0903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7353fff700 1 -- 172.21.15.194:0/1161101507 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877733821 0 0) 0x7f734c001110 con 0x7f735c05e230 2024-04-26T09:35:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7353fff700 1 -- 172.21.15.194:0/1161101507 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73480033f0 con 0x7f735c05e230 2024-04-26T09:35:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7361c57700 1 -- 172.21.15.194:0/1161101507 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 msgr2=0x7f735c0903d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7361c57700 1 --2- 172.21.15.194:0/1161101507 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 0x7f735c0903d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7361c57700 1 -- 172.21.15.194:0/1161101507 >> v1:172.21.15.194:6789/0 conn(0x7f735c05e230 legacy=0x7f735c0908c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7361c57700 1 -- 172.21.15.194:0/1161101507 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c0668a0 con 0x7f735c090fd0 2024-04-26T09:35:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7361c57700 1 --2- 172.21.15.194:0/1161101507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f735c090fd0 0x7f735c12cc50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f7354009e90 tx=0x7f7354009780).ready entity=mon.0 client_cookie=ff0970b50d6b4cb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7353fff700 1 -- 172.21.15.194:0/1161101507 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735400e070 con 0x7f735c090fd0 2024-04-26T09:35:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7353fff700 1 -- 172.21.15.194:0/1161101507 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735400ba70 con 0x7f735c090fd0 2024-04-26T09:35:42.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.084+0000 7f7353fff700 1 -- 172.21.15.194:0/1161101507 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735400bdf0 con 0x7f735c090fd0 2024-04-26T09:35:42.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.085+0000 7f73636ba700 1 -- 172.21.15.194:0/1161101507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f735c090fd0 msgr2=0x7f735c12cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.085+0000 7f73636ba700 1 --2- 172.21.15.194:0/1161101507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f735c090fd0 0x7f735c12cc50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f7354009e90 tx=0x7f7354009780).stop 2024-04-26T09:35:42.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.085+0000 7f73636ba700 1 -- 172.21.15.194:0/1161101507 shutdown_connections 2024-04-26T09:35:42.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.085+0000 7f73636ba700 1 --2- 172.21.15.194:0/1161101507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f735c090fd0 0x7f735c12cc50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.085+0000 7f73636ba700 1 --2- 172.21.15.194:0/1161101507 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 0x7f735c0903d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.085+0000 7f73636ba700 1 -- 172.21.15.194:0/1161101507 >> 172.21.15.194:0/1161101507 conn(0x7f735c0914c0 msgr2=0x7f735c0918d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:42.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.086+0000 7f73636ba700 1 -- 172.21.15.194:0/1161101507 shutdown_connections 2024-04-26T09:35:42.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.086+0000 7f73636ba700 1 -- 172.21.15.194:0/1161101507 wait complete. 2024-04-26T09:35:42.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.087+0000 7f73636ba700 1 Processor -- start 2024-04-26T09:35:42.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.087+0000 7f73636ba700 1 -- start start 2024-04-26T09:35:42.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.087+0000 7f73636ba700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f735c05e230 0x7f735c11a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.087+0000 7f73636ba700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 0x7f735c11a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.087+0000 7f73636ba700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f735c090fd0 0x7f735c11ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.087+0000 7f73636ba700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c11dc50 con 0x7f735c05e230 2024-04-26T09:35:42.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.087+0000 7f73636ba700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c11d030 con 0x7f735c08ecb0 2024-04-26T09:35:42.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.087+0000 7f73636ba700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f735c11d270 con 0x7f735c090fd0 2024-04-26T09:35:42.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7361c57700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f735c090fd0 0x7f735c11ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7360c55700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 0x7f735c11a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7360c55700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 0x7f735c11a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56822/0 (socket says 172.21.15.194:56822) 2024-04-26T09:35:42.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7361c57700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f735c090fd0 0x7f735c11ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:33032/0 (socket says 172.21.15.194:33032) 2024-04-26T09:35:42.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7360c55700 1 -- 172.21.15.194:0/2069731416 learned_addr learned my addr 172.21.15.194:0/2069731416 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7361456700 1 --2- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f735c05e230 0x7f735c11a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7361c57700 1 -- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 msgr2=0x7f735c11a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7361c57700 1 --2- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 0x7f735c11a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7361c57700 1 -- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f735c05e230 msgr2=0x7f735c11a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7361c57700 1 --2- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f735c05e230 0x7f735c11a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7361c57700 1 -- 172.21.15.194:0/2069731416 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7354009a20 con 0x7f735c090fd0 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7361c57700 1 --2- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f735c090fd0 0x7f735c11ae00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f735c06cc50 tx=0x7f7354009780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7360c55700 1 --2- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 0x7f735c11a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7351ffb700 1 -- 172.21.15.194:0/2069731416 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7354009440 con 0x7f735c090fd0 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7351ffb700 1 -- 172.21.15.194:0/2069731416 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735401d630 con 0x7f735c090fd0 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f73636ba700 1 -- 172.21.15.194:0/2069731416 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735c11bfd0 con 0x7f735c090fd0 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.088+0000 7f7351ffb700 1 -- 172.21.15.194:0/2069731416 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735401d9f0 con 0x7f735c090fd0 2024-04-26T09:35:42.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.089+0000 7f73636ba700 1 -- 172.21.15.194:0/2069731416 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735c11d890 con 0x7f735c090fd0 2024-04-26T09:35:42.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.090+0000 7f7351ffb700 1 -- 172.21.15.194:0/2069731416 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f7354003040 con 0x7f735c090fd0 2024-04-26T09:35:42.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.090+0000 7f7351ffb700 1 -- 172.21.15.194:0/2069731416 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f7354056ef0 con 0x7f735c090fd0 2024-04-26T09:35:42.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.090+0000 7f73636ba700 1 -- 172.21.15.194:0/2069731416 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735c091fe0 con 0x7f735c090fd0 2024-04-26T09:35:42.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.093+0000 7f4aab586700 1 Processor -- start 2024-04-26T09:35:42.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.094+0000 7f4aab586700 1 -- start start 2024-04-26T09:35:42.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.094+0000 7f4aab586700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f4aa4065de0 0x7f4aa40661c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.094+0000 7f4aab586700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa4122630 0x7f4aa4122a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.094+0000 7f4aab586700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aa4069840 con 0x7f4aa40666b0 2024-04-26T09:35:42.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.094+0000 7f4aab586700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aa4066f00 con 0x7f4aa4065de0 2024-04-26T09:35:42.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.094+0000 7f4aab586700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40691e0 con 0x7f4aa4122630 2024-04-26T09:35:42.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4aa9b23700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa4122630 0x7f4aa4122a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4aa8b21700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f4aa40666b0 0x7f4aa4121f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:51696/0 (socket says 172.21.15.194:51696) 2024-04-26T09:35:42.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4aa8b21700 1 -- 172.21.15.194:0/1221504809 learned_addr learned my addr 172.21.15.194:0/1221504809 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:35:42.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4a9bfff700 1 -- 172.21.15.194:0/1221504809 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266819847 0 0) 0x7f4aa4069840 con 0x7f4aa40666b0 2024-04-26T09:35:42.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4aa9b23700 1 -- 172.21.15.194:0/1221504809 >> v2:172.21.15.194:3300/0 conn(0x7f4aa4065de0 msgr2=0x7f4aa40661c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:42.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4aa9b23700 1 --2- 172.21.15.194:0/1221504809 >> v2:172.21.15.194:3300/0 conn(0x7f4aa4065de0 0x7f4aa40661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:42.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4aa9b23700 1 -- 172.21.15.194:0/1221504809 >> v1:172.21.15.194:6789/0 conn(0x7f4aa40666b0 legacy=0x7f4aa4121f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4aa9b23700 1 -- 172.21.15.194:0/1221504809 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa40694f0 con 0x7f4aa4122630 2024-04-26T09:35:42.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4aa9b23700 1 --2- 172.21.15.194:0/1221504809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa4122630 0x7f4aa4122a70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4a9c009e60 tx=0x7f4a9c009780).ready entity=mon.1 client_cookie=a9743dfe7755e213 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:42.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4a9bfff700 1 -- 172.21.15.194:0/1221504809 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c00e070 con 0x7f4aa4122630 2024-04-26T09:35:42.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4a9bfff700 1 -- 172.21.15.194:0/1221504809 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9c00bc30 con 0x7f4aa4122630 2024-04-26T09:35:42.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.095+0000 7f4a9bfff700 1 -- 172.21.15.194:0/1221504809 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c012410 con 0x7f4aa4122630 2024-04-26T09:35:42.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.096+0000 7f4aab586700 1 -- 172.21.15.194:0/1221504809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa4122630 msgr2=0x7f4aa4122a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.096+0000 7f4aab586700 1 --2- 172.21.15.194:0/1221504809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa4122630 0x7f4aa4122a70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4a9c009e60 tx=0x7f4a9c009780).stop 2024-04-26T09:35:42.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.096+0000 7f4aab586700 1 -- 172.21.15.194:0/1221504809 shutdown_connections 2024-04-26T09:35:42.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.096+0000 7f4aab586700 1 --2- 172.21.15.194:0/1221504809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa4122630 0x7f4aa4122a70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.096+0000 7f4aab586700 1 --2- 172.21.15.194:0/1221504809 >> v2:172.21.15.194:3300/0 conn(0x7f4aa4065de0 0x7f4aa40661c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2024-04-26T09:35:42.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.096+0000 7f4aab586700 1 -- 172.21.15.194:0/1221504809 >> 172.21.15.194:0/1221504809 conn(0x7f4aa4095750 msgr2=0x7f4aa4095b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:42.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.096+0000 7f4aab586700 1 -- 172.21.15.194:0/1221504809 shutdown_connections 2024-04-26T09:35:42.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.096+0000 7f4aab586700 1 -- 172.21.15.194:0/1221504809 wait complete. 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aab586700 1 Processor -- start 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aab586700 1 -- start start 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aab586700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4aa4065de0 0x7f4aa4122fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aab586700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa40666b0 0x7f4aa41234b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aab586700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4aa4122630 0x7f4aa41239a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aab586700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa412d100 con 0x7f4aa4122630 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aab586700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa4124790 con 0x7f4aa40666b0 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aab586700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa4125010 con 0x7f4aa4065de0 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aa8b21700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa40666b0 0x7f4aa41234b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aa8b21700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa40666b0 0x7f4aa41234b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56836/0 (socket says 172.21.15.194:56836) 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aa8b21700 1 -- 172.21.15.194:0/4223627093 learned_addr learned my addr 172.21.15.194:0/4223627093 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.097+0000 7f4aa9b23700 1 --2- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4aa4122630 0x7f4aa41239a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4aa9322700 1 --2- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4aa4065de0 0x7f4aa4122fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:42.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f7351ffb700 1 -- 172.21.15.194:0/2069731416 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f735403b410 con 0x7f735c090fd0 2024-04-26T09:35:42.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4aa9b23700 1 -- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4aa4065de0 msgr2=0x7f4aa4122fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4aa9b23700 1 --2- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4aa4065de0 0x7f4aa4122fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4aa9b23700 1 -- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa40666b0 msgr2=0x7f4aa41234b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4aa9b23700 1 --2- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa40666b0 0x7f4aa41234b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4aa9b23700 1 -- 172.21.15.194:0/4223627093 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c009a20 con 0x7f4aa4122630 2024-04-26T09:35:42.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4aa8b21700 1 --2- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa40666b0 0x7f4aa41234b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:42.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4aa9b23700 1 --2- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4aa4122630 0x7f4aa41239a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f4a9c014000 tx=0x7f4a9c009780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:42.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4a99ffb700 1 -- 172.21.15.194:0/4223627093 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c01c660 con 0x7f4aa4122630 2024-04-26T09:35:42.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4a99ffb700 1 -- 172.21.15.194:0/4223627093 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9c01c7c0 con 0x7f4aa4122630 2024-04-26T09:35:42.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4a99ffb700 1 -- 172.21.15.194:0/4223627093 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9c01cb80 con 0x7f4aa4122630 2024-04-26T09:35:42.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.098+0000 7f4aab586700 1 -- 172.21.15.194:0/4223627093 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa41249d0 con 0x7f4aa4122630 2024-04-26T09:35:42.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.099+0000 7f4aab586700 1 -- 172.21.15.194:0/4223627093 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa411f3c0 con 0x7f4aa4122630 2024-04-26T09:35:42.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.100+0000 7f4aab586700 1 -- 172.21.15.194:0/4223627093 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa4092a10 con 0x7f4aa4122630 2024-04-26T09:35:42.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.100+0000 7f4a99ffb700 1 -- 172.21.15.194:0/4223627093 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f4a9c027460 con 0x7f4aa4122630 2024-04-26T09:35:42.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.100+0000 7f4a99ffb700 1 -- 172.21.15.194:0/4223627093 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f4a9c057070 con 0x7f4aa4122630 2024-04-26T09:35:42.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.107+0000 7f4a99ffb700 1 -- 172.21.15.194:0/4223627093 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f4a9c03b6b0 con 0x7f4aa4122630 2024-04-26T09:35:42.174 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.170+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1040266975 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f2c04123940 con 0x7f2c04069ab0 2024-04-26T09:35:42.174 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.171+0000 7f2c027fc700 1 -- 172.21.15.194:0/1040266975 <== mon.1 v2:172.21.15.194:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2bf80396b0 con 0x7f2c04069ab0 2024-04-26T09:35:42.175 INFO:teuthology.orchestra.run.smithi194.stdout:38654705665 2024-04-26T09:35:42.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.172+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1040266975 >> v1:172.21.15.194:6816/72244 conn(0x7f2bf0041b80 legacy=0x7f2bf0044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.172+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04069ab0 msgr2=0x7f2c041191f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.172+0000 7f2c0bce7700 1 --2- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04069ab0 0x7f2c041191f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f2c040640b0 tx=0x7f2bf8009060).stop 2024-04-26T09:35:42.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.172+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1040266975 shutdown_connections 2024-04-26T09:35:42.177 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.172+0000 7f2c0bce7700 1 --2- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c04120750 0x7f2c041196e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.177 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.172+0000 7f2c0bce7700 1 --2- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f2c04063900 0x7f2c04118d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.177 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.172+0000 7f2c0bce7700 1 --2- 172.21.15.194:0/1040266975 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f2c04069ab0 0x7f2c041191f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.177 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.173+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1040266975 >> 172.21.15.194:0/1040266975 conn(0x7f2c04095750 msgr2=0x7f2c04120e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:42.177 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.173+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1040266975 shutdown_connections 2024-04-26T09:35:42.177 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.173+0000 7f2c0bce7700 1 -- 172.21.15.194:0/1040266975 wait complete. 2024-04-26T09:35:42.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.183+0000 7ffb6507c700 1 -- 172.21.15.194:0/3332195121 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ffb601134f0 con 0x7ffb60066d30 2024-04-26T09:35:42.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.183+0000 7ffb477fe700 1 -- 172.21.15.194:0/3332195121 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ffb50020070 con 0x7ffb60066d30 2024-04-26T09:35:42.187 INFO:teuthology.orchestra.run.smithi194.stdout:38654705666 2024-04-26T09:35:42.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.1 2024-04-26T09:35:42.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.185+0000 7ffb6507c700 1 -- 172.21.15.194:0/3332195121 >> v1:172.21.15.194:6816/72244 conn(0x7ffb4c041b80 legacy=0x7ffb4c044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.185+0000 7ffb6507c700 1 -- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb60066d30 msgr2=0x7ffb60111970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.185+0000 7ffb6507c700 1 --2- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb60066d30 0x7ffb60111970 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7ffb50014000 tx=0x7ffb50017000).stop 2024-04-26T09:35:42.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.185+0000 7ffb6507c700 1 -- 172.21.15.194:0/3332195121 shutdown_connections 2024-04-26T09:35:42.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.185+0000 7ffb6507c700 1 --2- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ffb60120320 0x7ffb60112350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.185+0000 7ffb6507c700 1 --2- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb60066d30 0x7ffb60111970 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.185+0000 7ffb6507c700 1 --2- 172.21.15.194:0/3332195121 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ffb60067600 0x7ffb60111e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.185+0000 7ffb6507c700 1 -- 172.21.15.194:0/3332195121 >> 172.21.15.194:0/3332195121 conn(0x7ffb60096860 msgr2=0x7ffb60122fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:42.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.185+0000 7ffb6507c700 1 -- 172.21.15.194:0/3332195121 shutdown_connections 2024-04-26T09:35:42.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.185+0000 7ffb6507c700 1 -- 172.21.15.194:0/3332195121 wait complete. 2024-04-26T09:35:42.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.3 2024-04-26T09:35:42.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.248+0000 7f73636ba700 1 -- 172.21.15.194:0/2069731416 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f735c068470 con 0x7f735c090fd0 2024-04-26T09:35:42.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.248+0000 7f7351ffb700 1 -- 172.21.15.194:0/2069731416 <== mon.2 v2:172.21.15.194:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7354035090 con 0x7f735c090fd0 2024-04-26T09:35:42.252 INFO:teuthology.orchestra.run.smithi194.stdout:38654705665 2024-04-26T09:35:42.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.250+0000 7f73636ba700 1 -- 172.21.15.194:0/2069731416 >> v1:172.21.15.194:6816/72244 conn(0x7f7340041b80 legacy=0x7f7340044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.250+0000 7f73636ba700 1 -- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f735c090fd0 msgr2=0x7f735c11ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.250+0000 7f73636ba700 1 --2- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f735c090fd0 0x7f735c11ae00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f735c06cc50 tx=0x7f7354009780).stop 2024-04-26T09:35:42.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.250+0000 7f73636ba700 1 -- 172.21.15.194:0/2069731416 shutdown_connections 2024-04-26T09:35:42.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.250+0000 7f73636ba700 1 --2- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f735c090fd0 0x7f735c11ae00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.250+0000 7f73636ba700 1 --2- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f735c05e230 0x7f735c11a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.250+0000 7f73636ba700 1 --2- 172.21.15.194:0/2069731416 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f735c08ecb0 0x7f735c11a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.250+0000 7f73636ba700 1 -- 172.21.15.194:0/2069731416 >> 172.21.15.194:0/2069731416 conn(0x7f735c0914c0 msgr2=0x7f735c0691f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:42.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.250+0000 7f73636ba700 1 -- 172.21.15.194:0/2069731416 shutdown_connections 2024-04-26T09:35:42.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.250+0000 7f73636ba700 1 -- 172.21.15.194:0/2069731416 wait complete. 2024-04-26T09:35:42.270 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705665 for osd.2 2024-04-26T09:35:42.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.296+0000 7f4aab586700 1 -- 172.21.15.194:0/4223627093 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4aa4068460 con 0x7f4aa4122630 2024-04-26T09:35:42.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.297+0000 7f4a99ffb700 1 -- 172.21.15.194:0/4223627093 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4a9c03b2c0 con 0x7f4aa4122630 2024-04-26T09:35:42.300 INFO:teuthology.orchestra.run.smithi194.stdout:38654705666 2024-04-26T09:35:42.302 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.298+0000 7f4aab586700 1 -- 172.21.15.194:0/4223627093 >> v1:172.21.15.194:6816/72244 conn(0x7f4a88041bd0 legacy=0x7f4a88044090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.302 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.298+0000 7f4aab586700 1 -- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4aa4122630 msgr2=0x7f4aa41239a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:42.302 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.298+0000 7f4aab586700 1 --2- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4aa4122630 0x7f4aa41239a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f4a9c014000 tx=0x7f4a9c009780).stop 2024-04-26T09:35:42.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.298+0000 7f4aab586700 1 -- 172.21.15.194:0/4223627093 shutdown_connections 2024-04-26T09:35:42.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.298+0000 7f4aab586700 1 --2- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4aa4122630 0x7f4aa41239a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.298+0000 7f4aab586700 1 --2- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4aa4065de0 0x7f4aa4122fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.298+0000 7f4aab586700 1 --2- 172.21.15.194:0/4223627093 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4aa40666b0 0x7f4aa41234b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:42.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.299+0000 7f4aab586700 1 -- 172.21.15.194:0/4223627093 >> 172.21.15.194:0/4223627093 conn(0x7f4aa4095750 msgr2=0x7f4aa4117cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:42.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.299+0000 7f4aab586700 1 -- 172.21.15.194:0/4223627093 shutdown_connections 2024-04-26T09:35:42.303 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:42.299+0000 7f4aab586700 1 -- 172.21.15.194:0/4223627093 wait complete. 2024-04-26T09:35:42.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705666 for osd.0 2024-04-26T09:35:43.189 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T09:35:43.201 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T09:35:43.271 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T09:35:43.311 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.308+0000 7fe9b5c74700 1 Processor -- start 2024-04-26T09:35:43.312 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.308+0000 7fe9b5c74700 1 -- start start 2024-04-26T09:35:43.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.308+0000 7fe9b5c74700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b0065de0 0x7fe9b00661c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.308+0000 7fe9b5c74700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fe9b00666b0 0x7fe9b0121f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.308+0000 7fe9b5c74700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0122410 0x7fe9b01228c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9b5c74700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b0069840 con 0x7fe9b00666b0 2024-04-26T09:35:43.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9b5c74700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b0066f00 con 0x7fe9b0122410 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9b5c74700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00691e0 con 0x7fe9b0065de0 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9af7fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b0065de0 0x7fe9b00661c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9af7fe700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b0065de0 0x7fe9b00661c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:33050/0 (socket says 172.21.15.194:33050) 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9af7fe700 1 -- 172.21.15.194:0/3384876893 learned_addr learned my addr 172.21.15.194:0/3384876893 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9affff700 1 --2- 172.21.15.194:0/3384876893 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0122410 0x7fe9b01228c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9aeffd700 1 --2- 172.21.15.194:0/3384876893 >> v2:172.21.15.194:3300/0 conn(0x7fe9b00666b0 0x7fe9b0121f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9af7fe700 1 -- 172.21.15.194:0/3384876893 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0122410 msgr2=0x7fe9b01228c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9af7fe700 1 --2- 172.21.15.194:0/3384876893 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0122410 0x7fe9b01228c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9af7fe700 1 -- 172.21.15.194:0/3384876893 >> v2:172.21.15.194:3300/0 conn(0x7fe9b00666b0 msgr2=0x7fe9b0121f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9af7fe700 1 --2- 172.21.15.194:0/3384876893 >> v2:172.21.15.194:3300/0 conn(0x7fe9b00666b0 0x7fe9b0121f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9af7fe700 1 -- 172.21.15.194:0/3384876893 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b00694f0 con 0x7fe9b0065de0 2024-04-26T09:35:43.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9aeffd700 1 --2- 172.21.15.194:0/3384876893 >> v2:172.21.15.194:3300/0 conn(0x7fe9b00666b0 0x7fe9b0121f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9af7fe700 1 --2- 172.21.15.194:0/3384876893 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b0065de0 0x7fe9b00661c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe9a0009830 tx=0x7fe9a00090f0).ready entity=mon.2 client_cookie=bc0ee16579a6e5ee server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9ae7fc700 1 -- 172.21.15.194:0/3384876893 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a000f510 con 0x7fe9b0065de0 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9ae7fc700 1 -- 172.21.15.194:0/3384876893 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a000f670 con 0x7fe9b0065de0 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.309+0000 7fe9ae7fc700 1 -- 172.21.15.194:0/3384876893 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a000fa10 con 0x7fe9b0065de0 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.310+0000 7fe9b5c74700 1 -- 172.21.15.194:0/3384876893 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b0065de0 msgr2=0x7fe9b00661c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.310+0000 7fe9b5c74700 1 --2- 172.21.15.194:0/3384876893 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b0065de0 0x7fe9b00661c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe9a0009830 tx=0x7fe9a00090f0).stop 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.310+0000 7fe9b5c74700 1 -- 172.21.15.194:0/3384876893 shutdown_connections 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.310+0000 7fe9b5c74700 1 --2- 172.21.15.194:0/3384876893 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0122410 0x7fe9b01228c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.310+0000 7fe9b5c74700 1 --2- 172.21.15.194:0/3384876893 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b0065de0 0x7fe9b00661c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.310+0000 7fe9b5c74700 1 --2- 172.21.15.194:0/3384876893 >> v2:172.21.15.194:3300/0 conn(0x7fe9b00666b0 0x7fe9b0121f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.310+0000 7fe9b5c74700 1 -- 172.21.15.194:0/3384876893 >> 172.21.15.194:0/3384876893 conn(0x7fe9b0095750 msgr2=0x7fe9b0095b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.310+0000 7fe9b5c74700 1 -- 172.21.15.194:0/3384876893 shutdown_connections 2024-04-26T09:35:43.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.310+0000 7fe9b5c74700 1 -- 172.21.15.194:0/3384876893 wait complete. 2024-04-26T09:35:43.317 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T09:35:43.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.311+0000 7fe9b5c74700 1 Processor -- start 2024-04-26T09:35:43.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.311+0000 7fe9b5c74700 1 -- start start 2024-04-26T09:35:43.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.312+0000 7fe9b5c74700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0065de0 0x7fe9b011e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.312+0000 7fe9b5c74700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b00666b0 0x7fe9b0115270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.312+0000 7fe9b5c74700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9b0122410 0x7fe9b0115760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.312+0000 7fe9b5c74700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b0116370 con 0x7fe9b0122410 2024-04-26T09:35:43.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.312+0000 7fe9b5c74700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b011ec90 con 0x7fe9b0065de0 2024-04-26T09:35:43.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.312+0000 7fe9b5c74700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b0117730 con 0x7fe9b00666b0 2024-04-26T09:35:43.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.312+0000 7fe9af7fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0065de0 0x7fe9b011e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.312+0000 7fe9af7fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0065de0 0x7fe9b011e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56858/0 (socket says 172.21.15.194:56858) 2024-04-26T09:35:43.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.312+0000 7fe9af7fe700 1 -- 172.21.15.194:0/78492153 learned_addr learned my addr 172.21.15.194:0/78492153 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:43.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.312+0000 7fe9aeffd700 1 --2- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b00666b0 0x7fe9b0115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.312+0000 7fe9affff700 1 --2- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9b0122410 0x7fe9b0115760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe9af7fe700 1 -- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b00666b0 msgr2=0x7fe9b0115270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe9af7fe700 1 --2- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b00666b0 0x7fe9b0115270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe9af7fe700 1 -- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9b0122410 msgr2=0x7fe9b0115760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe9af7fe700 1 --2- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9b0122410 0x7fe9b0115760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe9af7fe700 1 -- 172.21.15.194:0/78492153 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a40092a0 con 0x7fe9b0065de0 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe9affff700 1 --2- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9b0122410 0x7fe9b0115760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe9af7fe700 1 --2- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0065de0 0x7fe9b011e7a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe9a0008000 tx=0x7fe9a0015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe997fff700 1 -- 172.21.15.194:0/78492153 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0006910 con 0x7fe9b0065de0 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe997fff700 1 -- 172.21.15.194:0/78492153 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a0000bc0 con 0x7fe9b0065de0 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe9b5c74700 1 -- 172.21.15.194:0/78492153 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a0009370 con 0x7fe9b0065de0 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe997fff700 1 -- 172.21.15.194:0/78492153 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a000fa10 con 0x7fe9b0065de0 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.313+0000 7fe9b5c74700 1 -- 172.21.15.194:0/78492153 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b00644a0 con 0x7fe9b0065de0 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.314+0000 7fe9b5c74700 1 -- 172.21.15.194:0/78492153 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b0117b60 con 0x7fe9b0065de0 2024-04-26T09:35:43.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.314+0000 7fe997fff700 1 -- 172.21.15.194:0/78492153 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fe9a0021070 con 0x7fe9b0065de0 2024-04-26T09:35:43.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.315+0000 7fe997fff700 1 -- 172.21.15.194:0/78492153 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fe9a0056a60 con 0x7fe9b0065de0 2024-04-26T09:35:43.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.318+0000 7fe997fff700 1 -- 172.21.15.194:0/78492153 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fe9a001cd00 con 0x7fe9b0065de0 2024-04-26T09:35:43.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.350+0000 7ff771656700 1 Processor -- start 2024-04-26T09:35:43.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.350+0000 7ff771656700 1 -- start start 2024-04-26T09:35:43.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.350+0000 7ff771656700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c068fb0 0x7ff76c069390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.350+0000 7ff771656700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7ff76c08fbb0 0x7ff76c08ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.350+0000 7ff771656700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff76c0679b0 con 0x7ff76c090480 2024-04-26T09:35:43.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.350+0000 7ff771656700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76c067430 con 0x7ff76c08fbb0 2024-04-26T09:35:43.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.350+0000 7ff771656700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c069880 con 0x7ff76c068fb0 2024-04-26T09:35:43.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff76affd700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c068fb0 0x7ff76c069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff76affd700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c068fb0 0x7ff76c069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:33066/0 (socket says 172.21.15.194:33066) 2024-04-26T09:35:43.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff76affd700 1 -- 172.21.15.194:0/489746916 learned_addr learned my addr 172.21.15.194:0/489746916 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:43.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff76a7fc700 1 --2- 172.21.15.194:0/489746916 >> v2:172.21.15.194:3300/0 conn(0x7ff76c08fbb0 0x7ff76c08ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff76affd700 1 -- 172.21.15.194:0/489746916 >> v2:172.21.15.194:3300/0 conn(0x7ff76c08fbb0 msgr2=0x7ff76c08ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:43.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff76affd700 1 --2- 172.21.15.194:0/489746916 >> v2:172.21.15.194:3300/0 conn(0x7ff76c08fbb0 0x7ff76c08ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff76affd700 1 -- 172.21.15.194:0/489746916 >> v1:172.21.15.194:6789/0 conn(0x7ff76c090480 legacy=0x7ff76c0908c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff76affd700 1 -- 172.21.15.194:0/489746916 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c0665a0 con 0x7ff76c068fb0 2024-04-26T09:35:43.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff76affd700 1 --2- 172.21.15.194:0/489746916 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c068fb0 0x7ff76c069390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff75c009810 tx=0x7ff75c009060).ready entity=mon.2 client_cookie=d2e2d1d386f34a71 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:43.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff769ffb700 1 -- 172.21.15.194:0/489746916 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c005880 con 0x7ff76c068fb0 2024-04-26T09:35:43.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff769ffb700 1 -- 172.21.15.194:0/489746916 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff75c0059e0 con 0x7ff76c068fb0 2024-04-26T09:35:43.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.351+0000 7ff769ffb700 1 -- 172.21.15.194:0/489746916 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c005d60 con 0x7ff76c068fb0 2024-04-26T09:35:43.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.352+0000 7ff771656700 1 -- 172.21.15.194:0/489746916 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c068fb0 msgr2=0x7ff76c069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.352+0000 7ff771656700 1 --2- 172.21.15.194:0/489746916 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c068fb0 0x7ff76c069390 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ff75c009810 tx=0x7ff75c009060).stop 2024-04-26T09:35:43.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.352+0000 7ff771656700 1 -- 172.21.15.194:0/489746916 shutdown_connections 2024-04-26T09:35:43.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.352+0000 7ff771656700 1 --2- 172.21.15.194:0/489746916 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c068fb0 0x7ff76c069390 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.359 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.352+0000 7ff771656700 1 --2- 172.21.15.194:0/489746916 >> v2:172.21.15.194:3300/0 conn(0x7ff76c08fbb0 0x7ff76c08ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.352+0000 7ff771656700 1 -- 172.21.15.194:0/489746916 >> 172.21.15.194:0/489746916 conn(0x7ff76c0947f0 msgr2=0x7ff76c094c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.352+0000 7ff771656700 1 -- 172.21.15.194:0/489746916 shutdown_connections 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.352+0000 7ff771656700 1 -- 172.21.15.194:0/489746916 wait complete. 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.353+0000 7ff771656700 1 Processor -- start 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.353+0000 7ff771656700 1 -- start start 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff771656700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff76c068fb0 0x7ff76c08d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff771656700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c08fbb0 0x7ff76c132130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff771656700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff76c090480 0x7ff76c1308c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff771656700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c08de80 con 0x7ff76c090480 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff771656700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c08ee90 con 0x7ff76c068fb0 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff771656700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c08f0d0 con 0x7ff76c08fbb0 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff76a7fc700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c08fbb0 0x7ff76c132130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff76a7fc700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c08fbb0 0x7ff76c132130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:33080/0 (socket says 172.21.15.194:33080) 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff76a7fc700 1 -- 172.21.15.194:0/1984541272 learned_addr learned my addr 172.21.15.194:0/1984541272 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:43.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff76b7fe700 1 --2- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff76c090480 0x7ff76c1308c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff76affd700 1 --2- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff76c068fb0 0x7ff76c08d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff76a7fc700 1 -- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff76c068fb0 msgr2=0x7ff76c08d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff76a7fc700 1 --2- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff76c068fb0 0x7ff76c08d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff76a7fc700 1 -- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff76c090480 msgr2=0x7ff76c1308c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff76a7fc700 1 --2- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff76c090480 0x7ff76c1308c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff76a7fc700 1 -- 172.21.15.194:0/1984541272 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff75c009300 con 0x7ff76c08fbb0 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff76a7fc700 1 --2- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c08fbb0 0x7ff76c132130 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff76000e950 tx=0x7ff760009060).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.354+0000 7ff7537fe700 1 -- 172.21.15.194:0/1984541272 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7600094f0 con 0x7ff76c08fbb0 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.355+0000 7ff7537fe700 1 -- 172.21.15.194:0/1984541272 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff760012600 con 0x7ff76c08fbb0 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.355+0000 7ff771656700 1 -- 172.21.15.194:0/1984541272 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76c119a20 con 0x7ff76c08fbb0 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.355+0000 7ff7537fe700 1 -- 172.21.15.194:0/1984541272 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7600129a0 con 0x7ff76c08fbb0 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.355+0000 7ff771656700 1 -- 172.21.15.194:0/1984541272 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76c132620 con 0x7ff76c08fbb0 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.356+0000 7ff771656700 1 -- 172.21.15.194:0/1984541272 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76c08dc90 con 0x7ff76c08fbb0 2024-04-26T09:35:43.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.356+0000 7ff7537fe700 1 -- 172.21.15.194:0/1984541272 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7ff760012d50 con 0x7ff76c08fbb0 2024-04-26T09:35:43.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.356+0000 7ff7537fe700 1 -- 172.21.15.194:0/1984541272 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7ff760055200 con 0x7ff76c08fbb0 2024-04-26T09:35:43.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.363+0000 7ff7537fe700 1 -- 172.21.15.194:0/1984541272 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7ff760035c00 con 0x7ff76c08fbb0 2024-04-26T09:35:43.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.397+0000 7efef36d0700 1 Processor -- start 2024-04-26T09:35:43.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.397+0000 7efef36d0700 1 -- start start 2024-04-26T09:35:43.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef36d0700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7efeec063900 0x7efeec0661d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef36d0700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec0666c0 0x7efeec066ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef36d0700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeec065da0 con 0x7efeec125250 2024-04-26T09:35:43.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef36d0700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeec0683e0 con 0x7efeec063900 2024-04-26T09:35:43.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef36d0700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7efeec067830 con 0x7efeec0666c0 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef0c6b700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec0666c0 0x7efeec066ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef146c700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7efeec063900 0x7efeec0661d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef0c6b700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec0666c0 0x7efeec066ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:33084/0 (socket says 172.21.15.194:33084) 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef0c6b700 1 -- 172.21.15.194:0/912632737 learned_addr learned my addr 172.21.15.194:0/912632737 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef0c6b700 1 -- 172.21.15.194:0/912632737 >> v2:172.21.15.194:3300/0 conn(0x7efeec063900 msgr2=0x7efeec0661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef0c6b700 1 --2- 172.21.15.194:0/912632737 >> v2:172.21.15.194:3300/0 conn(0x7efeec063900 0x7efeec0661d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef0c6b700 1 -- 172.21.15.194:0/912632737 >> v1:172.21.15.194:6789/0 conn(0x7efeec125250 legacy=0x7efeec125700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef0c6b700 1 -- 172.21.15.194:0/912632737 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec067fa0 con 0x7efeec0666c0 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.398+0000 7efef0c6b700 1 --2- 172.21.15.194:0/912632737 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec0666c0 0x7efeec066ac0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7efedc006fd0 tx=0x7efedc0090f0).ready entity=mon.2 client_cookie=91b9656a7e7f314e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.399+0000 7efee3fff700 1 -- 172.21.15.194:0/912632737 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc00b580 con 0x7efeec0666c0 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.399+0000 7efee3fff700 1 -- 172.21.15.194:0/912632737 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efedc00b6e0 con 0x7efeec0666c0 2024-04-26T09:35:43.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.399+0000 7efee3fff700 1 -- 172.21.15.194:0/912632737 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efedc00baa0 con 0x7efeec0666c0 2024-04-26T09:35:43.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.399+0000 7efef36d0700 1 -- 172.21.15.194:0/912632737 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec0666c0 msgr2=0x7efeec066ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.399+0000 7efef36d0700 1 --2- 172.21.15.194:0/912632737 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec0666c0 0x7efeec066ac0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7efedc006fd0 tx=0x7efedc0090f0).stop 2024-04-26T09:35:43.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.400+0000 7efef36d0700 1 -- 172.21.15.194:0/912632737 shutdown_connections 2024-04-26T09:35:43.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.400+0000 7efef36d0700 1 --2- 172.21.15.194:0/912632737 >> v2:172.21.15.194:3300/0 conn(0x7efeec063900 0x7efeec0661d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.400+0000 7efef36d0700 1 --2- 172.21.15.194:0/912632737 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec0666c0 0x7efeec066ac0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.400+0000 7efef36d0700 1 -- 172.21.15.194:0/912632737 >> 172.21.15.194:0/912632737 conn(0x7efeec095750 msgr2=0x7efeec095b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:43.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.400+0000 7efef36d0700 1 -- 172.21.15.194:0/912632737 shutdown_connections 2024-04-26T09:35:43.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.400+0000 7efef36d0700 1 -- 172.21.15.194:0/912632737 wait complete. 2024-04-26T09:35:43.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.401+0000 7efef36d0700 1 Processor -- start 2024-04-26T09:35:43.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.401+0000 7efef36d0700 1 -- start start 2024-04-26T09:35:43.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.401+0000 7efef36d0700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efeec063900 0x7efeec1174d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.401+0000 7efef36d0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeec0666c0 0x7efeec1179c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.401+0000 7efef36d0700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec125250 0x7efeec117eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.401+0000 7efef36d0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec11ab90 con 0x7efeec0666c0 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.401+0000 7efef36d0700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeec11a210 con 0x7efeec063900 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.401+0000 7efef36d0700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7efeec122d80 con 0x7efeec125250 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef146c700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efeec063900 0x7efeec1174d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef146c700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efeec063900 0x7efeec1174d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56870/0 (socket says 172.21.15.194:56870) 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef146c700 1 -- 172.21.15.194:0/1140345394 learned_addr learned my addr 172.21.15.194:0/1140345394 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef1c6d700 1 --2- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec125250 0x7efeec117eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef0c6b700 1 --2- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeec0666c0 0x7efeec1179c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef146c700 1 -- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec125250 msgr2=0x7efeec117eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef146c700 1 --2- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec125250 0x7efeec117eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef146c700 1 -- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeec0666c0 msgr2=0x7efeec1179c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef146c700 1 --2- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeec0666c0 0x7efeec1179c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef146c700 1 -- 172.21.15.194:0/1140345394 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efedc009370 con 0x7efeec063900 2024-04-26T09:35:43.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef146c700 1 --2- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efeec063900 0x7efeec1174d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7efee800e930 tx=0x7efee8009060).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:43.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efee1ffb700 1 -- 172.21.15.194:0/1140345394 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8009510 con 0x7efeec063900 2024-04-26T09:35:43.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efee1ffb700 1 -- 172.21.15.194:0/1140345394 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee8012600 con 0x7efeec063900 2024-04-26T09:35:43.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efef36d0700 1 -- 172.21.15.194:0/1140345394 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeec11adc0 con 0x7efeec063900 2024-04-26T09:35:43.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.402+0000 7efee1ffb700 1 -- 172.21.15.194:0/1140345394 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8012980 con 0x7efeec063900 2024-04-26T09:35:43.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.403+0000 7efef36d0700 1 -- 172.21.15.194:0/1140345394 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeec1183a0 con 0x7efeec063900 2024-04-26T09:35:43.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.404+0000 7efef36d0700 1 -- 172.21.15.194:0/1140345394 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeec092140 con 0x7efeec063900 2024-04-26T09:35:43.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.404+0000 7efee1ffb700 1 -- 172.21.15.194:0/1140345394 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7efee801b400 con 0x7efeec063900 2024-04-26T09:35:43.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.404+0000 7efee1ffb700 1 -- 172.21.15.194:0/1140345394 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7efee805c360 con 0x7efeec063900 2024-04-26T09:35:43.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.411+0000 7efee1ffb700 1 -- 172.21.15.194:0/1140345394 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7efee80351e0 con 0x7efeec063900 2024-04-26T09:35:43.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.444+0000 7fa65397b700 1 Processor -- start 2024-04-26T09:35:43.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa65397b700 1 -- start start 2024-04-26T09:35:43.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa65397b700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 0x7fa64c067110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa65397b700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fa64c067600 0x7fa64c11fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa65397b700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c120100 0x7fa64c1205b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa65397b700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa64c068740 con 0x7fa64c067600 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa65397b700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c069910 con 0x7fa64c066d30 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa65397b700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c120aa0 con 0x7fa64c120100 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa651f18700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c120100 0x7fa64c1205b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa650f16700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fa64c067600 0x7fa64c11fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa651f18700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c120100 0x7fa64c1205b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:33106/0 (socket says 172.21.15.194:33106) 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa651f18700 1 -- 172.21.15.194:0/3913838447 learned_addr learned my addr 172.21.15.194:0/3913838447 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.445+0000 7fa651717700 1 --2- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 0x7fa64c067110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.446+0000 7fa650f16700 1 -- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c120100 msgr2=0x7fa64c1205b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.446+0000 7fa650f16700 1 --2- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c120100 0x7fa64c1205b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.446+0000 7fa650f16700 1 -- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 msgr2=0x7fa64c067110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:43.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.446+0000 7fa650f16700 1 --2- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 0x7fa64c067110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.446+0000 7fa650f16700 1 -- 172.21.15.194:0/3913838447 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c069650 con 0x7fa64c067600 2024-04-26T09:35:43.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.446+0000 7fa651f18700 1 --2- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c120100 0x7fa64c1205b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:43.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.446+0000 7fa651717700 1 --2- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 0x7fa64c067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:43.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.446+0000 7fa650f16700 1 --2- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa64c067600 0x7fa64c11fc10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa63c009760 tx=0x7fa63c009060).ready entity=mon.0 client_cookie=f81fc611fb8e47da server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:43.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.446+0000 7fa643fff700 1 -- 172.21.15.194:0/3913838447 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c0090d0 con 0x7fa64c067600 2024-04-26T09:35:43.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.446+0000 7fa643fff700 1 -- 172.21.15.194:0/3913838447 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63c00d4a0 con 0x7fa64c067600 2024-04-26T09:35:43.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.446+0000 7fa643fff700 1 -- 172.21.15.194:0/3913838447 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c00d820 con 0x7fa64c067600 2024-04-26T09:35:43.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.447+0000 7fa65397b700 1 -- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa64c067600 msgr2=0x7fa64c11fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.447+0000 7fa65397b700 1 --2- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa64c067600 0x7fa64c11fc10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa63c009760 tx=0x7fa63c009060).stop 2024-04-26T09:35:43.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.447+0000 7fa65397b700 1 -- 172.21.15.194:0/3913838447 shutdown_connections 2024-04-26T09:35:43.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.447+0000 7fa65397b700 1 --2- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c120100 0x7fa64c1205b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.447+0000 7fa65397b700 1 --2- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 0x7fa64c067110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.447+0000 7fa65397b700 1 --2- 172.21.15.194:0/3913838447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa64c067600 0x7fa64c11fc10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.447+0000 7fa65397b700 1 -- 172.21.15.194:0/3913838447 >> 172.21.15.194:0/3913838447 conn(0x7fa64c096860 msgr2=0x7fa64c096c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.447+0000 7fa65397b700 1 -- 172.21.15.194:0/3913838447 shutdown_connections 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.447+0000 7fa65397b700 1 -- 172.21.15.194:0/3913838447 wait complete. 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.448+0000 7fa65397b700 1 Processor -- start 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.448+0000 7fa65397b700 1 -- start start 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa65397b700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 0x7fa64c113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa65397b700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c067600 0x7fa64c114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa65397b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa64c120100 0x7fa64c1144f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa65397b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c1149e0 con 0x7fa64c120100 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa65397b700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c115700 con 0x7fa64c066d30 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa65397b700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa64c11fd40 con 0x7fa64c067600 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa651f18700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa64c120100 0x7fa64c1144f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa651f18700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa64c120100 0x7fa64c1144f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48374/0 (socket says 172.21.15.194:48374) 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa651f18700 1 -- 172.21.15.194:0/1449676717 learned_addr learned my addr 172.21.15.194:0/1449676717 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa650f16700 1 --2- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c067600 0x7fa64c114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa651717700 1 --2- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 0x7fa64c113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa651f18700 1 -- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c067600 msgr2=0x7fa64c114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa651f18700 1 --2- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c067600 0x7fa64c114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa651f18700 1 -- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 msgr2=0x7fa64c113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa651f18700 1 --2- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 0x7fa64c113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa651f18700 1 -- 172.21.15.194:0/1449676717 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6480092a0 con 0x7fa64c120100 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.449+0000 7fa651717700 1 --2- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 0x7fa64c113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.450+0000 7fa651f18700 1 --2- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa64c120100 0x7fa64c1144f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fa644009fd0 tx=0x7fa64400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.450+0000 7fa641ffb700 1 -- 172.21.15.194:0/1449676717 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6440092e0 con 0x7fa64c120100 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.450+0000 7fa641ffb700 1 -- 172.21.15.194:0/1449676717 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64400f040 con 0x7fa64c120100 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.450+0000 7fa641ffb700 1 -- 172.21.15.194:0/1449676717 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64400cbd0 con 0x7fa64c120100 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.450+0000 7fa65397b700 1 -- 172.21.15.194:0/1449676717 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa644009ae0 con 0x7fa64c120100 2024-04-26T09:35:43.456 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.450+0000 7fa65397b700 1 -- 172.21.15.194:0/1449676717 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63c0092a0 con 0x7fa64c120100 2024-04-26T09:35:43.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.452+0000 7fa65397b700 1 -- 172.21.15.194:0/1449676717 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c092a10 con 0x7fa64c120100 2024-04-26T09:35:43.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.453+0000 7fa641ffb700 1 -- 172.21.15.194:0/1449676717 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fa644007400 con 0x7fa64c120100 2024-04-26T09:35:43.457 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.453+0000 7fa641ffb700 1 -- 172.21.15.194:0/1449676717 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fa6440552c0 con 0x7fa64c120100 2024-04-26T09:35:43.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.459+0000 7fa641ffb700 1 -- 172.21.15.194:0/1449676717 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fa644035c60 con 0x7fa64c120100 2024-04-26T09:35:43.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.490+0000 7fe9b5c74700 1 -- 172.21.15.194:0/78492153 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe9b0046ce0 con 0x7fe9b0065de0 2024-04-26T09:35:43.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.491+0000 7fe997fff700 1 -- 172.21.15.194:0/78492153 <== mon.1 v2:172.21.15.194:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe9a001cb30 con 0x7fe9b0065de0 2024-04-26T09:35:43.495 INFO:teuthology.orchestra.run.smithi194.stdout:38654705666 2024-04-26T09:35:43.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.492+0000 7fe9b5c74700 1 -- 172.21.15.194:0/78492153 >> v1:172.21.15.194:6816/72244 conn(0x7fe99c041b80 legacy=0x7fe99c044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.493+0000 7fe9b5c74700 1 -- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0065de0 msgr2=0x7fe9b011e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.493+0000 7fe9b5c74700 1 --2- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0065de0 0x7fe9b011e7a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe9a0008000 tx=0x7fe9a0015000).stop 2024-04-26T09:35:43.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.493+0000 7fe9b5c74700 1 -- 172.21.15.194:0/78492153 shutdown_connections 2024-04-26T09:35:43.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.493+0000 7fe9b5c74700 1 --2- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9b0122410 0x7fe9b0115760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.493+0000 7fe9b5c74700 1 --2- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fe9b0065de0 0x7fe9b011e7a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.493+0000 7fe9b5c74700 1 --2- 172.21.15.194:0/78492153 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fe9b00666b0 0x7fe9b0115270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.493+0000 7fe9b5c74700 1 -- 172.21.15.194:0/78492153 >> 172.21.15.194:0/78492153 conn(0x7fe9b0095750 msgr2=0x7fe9b011f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:43.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.493+0000 7fe9b5c74700 1 -- 172.21.15.194:0/78492153 shutdown_connections 2024-04-26T09:35:43.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.493+0000 7fe9b5c74700 1 -- 172.21.15.194:0/78492153 wait complete. 2024-04-26T09:35:43.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.1 2024-04-26T09:35:43.513 DEBUG:teuthology.parallel:result is None 2024-04-26T09:35:43.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.530+0000 7ff771656700 1 -- 172.21.15.194:0/1984541272 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7ff76c095310 con 0x7ff76c08fbb0 2024-04-26T09:35:43.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.530+0000 7ff7537fe700 1 -- 172.21.15.194:0/1984541272 <== mon.2 v2:172.21.15.194:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff760016030 con 0x7ff76c08fbb0 2024-04-26T09:35:43.534 INFO:teuthology.orchestra.run.smithi194.stdout:38654705667 2024-04-26T09:35:43.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.532+0000 7ff771656700 1 -- 172.21.15.194:0/1984541272 >> v1:172.21.15.194:6816/72244 conn(0x7ff758041b80 legacy=0x7ff758044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.532+0000 7ff771656700 1 -- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c08fbb0 msgr2=0x7ff76c132130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.532+0000 7ff771656700 1 --2- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c08fbb0 0x7ff76c132130 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff76000e950 tx=0x7ff760009060).stop 2024-04-26T09:35:43.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.532+0000 7ff771656700 1 -- 172.21.15.194:0/1984541272 shutdown_connections 2024-04-26T09:35:43.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.532+0000 7ff771656700 1 --2- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff76c090480 0x7ff76c1308c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.532+0000 7ff771656700 1 --2- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff76c068fb0 0x7ff76c08d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.532+0000 7ff771656700 1 --2- 172.21.15.194:0/1984541272 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff76c08fbb0 0x7ff76c132130 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.532+0000 7ff771656700 1 -- 172.21.15.194:0/1984541272 >> 172.21.15.194:0/1984541272 conn(0x7ff76c0947f0 msgr2=0x7ff76c085120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:43.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.532+0000 7ff771656700 1 -- 172.21.15.194:0/1984541272 shutdown_connections 2024-04-26T09:35:43.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.532+0000 7ff771656700 1 -- 172.21.15.194:0/1984541272 wait complete. 2024-04-26T09:35:43.547 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.3 2024-04-26T09:35:43.548 DEBUG:teuthology.parallel:result is None 2024-04-26T09:35:43.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.563+0000 7efef36d0700 1 -- 172.21.15.194:0/1140345394 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7efeec092a10 con 0x7efeec063900 2024-04-26T09:35:43.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.564+0000 7efee1ffb700 1 -- 172.21.15.194:0/1140345394 <== mon.1 v2:172.21.15.194:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efee80389d0 con 0x7efeec063900 2024-04-26T09:35:43.567 INFO:teuthology.orchestra.run.smithi194.stdout:38654705666 2024-04-26T09:35:43.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.566+0000 7efef36d0700 1 -- 172.21.15.194:0/1140345394 >> v1:172.21.15.194:6816/72244 conn(0x7efed0041b80 legacy=0x7efed0044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.566+0000 7efef36d0700 1 -- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efeec063900 msgr2=0x7efeec1174d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.566+0000 7efef36d0700 1 --2- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efeec063900 0x7efeec1174d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7efee800e930 tx=0x7efee8009060).stop 2024-04-26T09:35:43.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.566+0000 7efef36d0700 1 -- 172.21.15.194:0/1140345394 shutdown_connections 2024-04-26T09:35:43.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.566+0000 7efef36d0700 1 --2- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efeec125250 0x7efeec117eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.566+0000 7efef36d0700 1 --2- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efeec063900 0x7efeec1174d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.566+0000 7efef36d0700 1 --2- 172.21.15.194:0/1140345394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeec0666c0 0x7efeec1179c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.566+0000 7efef36d0700 1 -- 172.21.15.194:0/1140345394 >> 172.21.15.194:0/1140345394 conn(0x7efeec095750 msgr2=0x7efeec123f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:43.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.566+0000 7efef36d0700 1 -- 172.21.15.194:0/1140345394 shutdown_connections 2024-04-26T09:35:43.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.566+0000 7efef36d0700 1 -- 172.21.15.194:0/1140345394 wait complete. 2024-04-26T09:35:43.581 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705666 got 38654705666 for osd.2 2024-04-26T09:35:43.581 DEBUG:teuthology.parallel:result is None 2024-04-26T09:35:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.609+0000 7fa65397b700 1 -- 172.21.15.194:0/1449676717 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa64c093270 con 0x7fa64c120100 2024-04-26T09:35:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.610+0000 7fa641ffb700 1 -- 172.21.15.194:0/1449676717 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa644039410 con 0x7fa64c120100 2024-04-26T09:35:43.613 INFO:teuthology.orchestra.run.smithi194.stdout:38654705667 2024-04-26T09:35:43.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.611+0000 7fa65397b700 1 -- 172.21.15.194:0/1449676717 >> v1:172.21.15.194:6816/72244 conn(0x7fa630041bd0 legacy=0x7fa630044090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.611+0000 7fa65397b700 1 -- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa64c120100 msgr2=0x7fa64c1144f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.611+0000 7fa65397b700 1 --2- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa64c120100 0x7fa64c1144f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7fa644009fd0 tx=0x7fa64400b040).stop 2024-04-26T09:35:43.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.611+0000 7fa65397b700 1 -- 172.21.15.194:0/1449676717 shutdown_connections 2024-04-26T09:35:43.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.611+0000 7fa65397b700 1 --2- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa64c120100 0x7fa64c1144f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.611+0000 7fa65397b700 1 --2- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa64c066d30 0x7fa64c113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.611+0000 7fa65397b700 1 --2- 172.21.15.194:0/1449676717 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa64c067600 0x7fa64c114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.612+0000 7fa65397b700 1 -- 172.21.15.194:0/1449676717 >> 172.21.15.194:0/1449676717 conn(0x7fa64c096860 msgr2=0x7fa64c122430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:43.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.612+0000 7fa65397b700 1 -- 172.21.15.194:0/1449676717 shutdown_connections 2024-04-26T09:35:43.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.612+0000 7fa65397b700 1 -- 172.21.15.194:0/1449676717 wait complete. 2024-04-26T09:35:43.626 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.0 2024-04-26T09:35:43.627 DEBUG:teuthology.parallel:result is None 2024-04-26T09:35:43.627 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T09:35:43.627 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T09:35:43.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f688104a700 1 Processor -- start 2024-04-26T09:35:43.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f688104a700 1 -- start start 2024-04-26T09:35:43.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f688104a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c05e230 0x7f687c0654a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f688104a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c12acb0 0x7f687c12d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f688104a700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687c06a6c0 con 0x7f687c065990 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f688104a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687c05e8c0 con 0x7f687c12acb0 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f688104a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f687c0665c0 con 0x7f687c05e230 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f687ad9d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c05e230 0x7f687c0654a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f687b59e700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c12acb0 0x7f687c12d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f687b59e700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c12acb0 0x7f687c12d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56892/0 (socket says 172.21.15.194:56892) 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f687ad9d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c05e230 0x7f687c0654a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:33112/0 (socket says 172.21.15.194:33112) 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.777+0000 7f687b59e700 1 -- 172.21.15.194:0/2426815080 learned_addr learned my addr 172.21.15.194:0/2426815080 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f6879d9b700 1 -- 172.21.15.194:0/2426815080 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248026976 0 0) 0x7f687c06a6c0 con 0x7f687c065990 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f687b59e700 1 -- 172.21.15.194:0/2426815080 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c05e230 msgr2=0x7f687c0654a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f687b59e700 1 --2- 172.21.15.194:0/2426815080 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c05e230 0x7f687c0654a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f687b59e700 1 -- 172.21.15.194:0/2426815080 >> v1:172.21.15.194:6789/0 conn(0x7f687c065990 legacy=0x7f687c065d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f687b59e700 1 -- 172.21.15.194:0/2426815080 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687c08eec0 con 0x7f687c12acb0 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f687ad9d700 1 --2- 172.21.15.194:0/2426815080 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c05e230 0x7f687c0654a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:43.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f687b59e700 1 --2- 172.21.15.194:0/2426815080 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c12acb0 0x7f687c12d0a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6874009f80 tx=0x7f6874009840).ready entity=mon.1 client_cookie=5985cf1659a7c26e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:43.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f6879d9b700 1 -- 172.21.15.194:0/2426815080 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687400e070 con 0x7f687c12acb0 2024-04-26T09:35:43.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f6879d9b700 1 -- 172.21.15.194:0/2426815080 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687400ba40 con 0x7f687c12acb0 2024-04-26T09:35:43.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f6879d9b700 1 -- 172.21.15.194:0/2426815080 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687400bdc0 con 0x7f687c12acb0 2024-04-26T09:35:43.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f688104a700 1 -- 172.21.15.194:0/2426815080 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c12acb0 msgr2=0x7f687c12d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f688104a700 1 --2- 172.21.15.194:0/2426815080 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c12acb0 0x7f687c12d0a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6874009f80 tx=0x7f6874009840).stop 2024-04-26T09:35:43.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f688104a700 1 -- 172.21.15.194:0/2426815080 shutdown_connections 2024-04-26T09:35:43.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f688104a700 1 --2- 172.21.15.194:0/2426815080 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c12acb0 0x7f687c12d0a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f688104a700 1 --2- 172.21.15.194:0/2426815080 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c05e230 0x7f687c0654a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f688104a700 1 -- 172.21.15.194:0/2426815080 >> 172.21.15.194:0/2426815080 conn(0x7f687c091910 msgr2=0x7f687c091d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:43.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f688104a700 1 -- 172.21.15.194:0/2426815080 shutdown_connections 2024-04-26T09:35:43.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.778+0000 7f688104a700 1 -- 172.21.15.194:0/2426815080 wait complete. 2024-04-26T09:35:43.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f688104a700 1 Processor -- start 2024-04-26T09:35:43.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f688104a700 1 -- start start 2024-04-26T09:35:43.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f688104a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f687c05e230 0x7f687c134cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f688104a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c065990 0x7f687c1351c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f688104a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c12acb0 0x7f687c1356b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f688104a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687c088b50 con 0x7f687c05e230 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f688104a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687c125d20 con 0x7f687c065990 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f688104a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f687c089120 con 0x7f687c12acb0 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f687ad9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f687c05e230 0x7f687c134cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f687ad9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f687c05e230 0x7f687c134cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48390/0 (socket says 172.21.15.194:48390) 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f687ad9d700 1 -- 172.21.15.194:0/1879288967 learned_addr learned my addr 172.21.15.194:0/1879288967 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f687b59e700 1 --2- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c12acb0 0x7f687c1356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f687a59c700 1 --2- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c065990 0x7f687c1351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f687ad9d700 1 -- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c12acb0 msgr2=0x7f687c1356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f687ad9d700 1 --2- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c12acb0 0x7f687c1356b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f687ad9d700 1 -- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c065990 msgr2=0x7f687c1351c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f687ad9d700 1 --2- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c065990 0x7f687c1351c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.780+0000 7f687ad9d700 1 -- 172.21.15.194:0/1879288967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686c009250 con 0x7f687c05e230 2024-04-26T09:35:43.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.781+0000 7f687ad9d700 1 --2- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f687c05e230 0x7f687c134cd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f686c00ec80 tx=0x7f686c006f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:43.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.781+0000 7f68637fe700 1 -- 172.21.15.194:0/1879288967 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c0096a0 con 0x7f687c05e230 2024-04-26T09:35:43.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.781+0000 7f68637fe700 1 -- 172.21.15.194:0/1879288967 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686c005880 con 0x7f687c05e230 2024-04-26T09:35:43.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.781+0000 7f688104a700 1 -- 172.21.15.194:0/1879288967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6874009ae0 con 0x7f687c05e230 2024-04-26T09:35:43.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.781+0000 7f68637fe700 1 -- 172.21.15.194:0/1879288967 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686c005c00 con 0x7f687c05e230 2024-04-26T09:35:43.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.781+0000 7f688104a700 1 -- 172.21.15.194:0/1879288967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f687c087960 con 0x7f687c05e230 2024-04-26T09:35:43.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.782+0000 7f68637fe700 1 -- 172.21.15.194:0/1879288967 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f686c00c7f0 con 0x7f687c05e230 2024-04-26T09:35:43.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.782+0000 7f68637fe700 1 -- 172.21.15.194:0/1879288967 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f686c0552c0 con 0x7f687c05e230 2024-04-26T09:35:43.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.782+0000 7f688104a700 1 -- 172.21.15.194:0/1879288967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687c08a5b0 con 0x7f687c05e230 2024-04-26T09:35:43.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.786+0000 7f68637fe700 1 -- 172.21.15.194:0/1879288967 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f687c08a5b0 con 0x7f687c05e230 2024-04-26T09:35:43.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.922+0000 7f688104a700 1 -- 172.21.15.194:0/1879288967 --> v1:172.21.15.194:6816/72244 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f687c08a5b0 con 0x7f6868041bb0 2024-04-26T09:35:43.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.924+0000 7f68637fe700 1 -- 172.21.15.194:0/1879288967 <== mgr.4111 v1:172.21.15.194:6816/72244 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+32744 (unknown 2965378022 0 3809072774) 0x7f687c08a5b0 con 0x7f6868041bb0 2024-04-26T09:35:43.928 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.925+0000 7f688104a700 1 -- 172.21.15.194:0/1879288967 >> v1:172.21.15.194:6816/72244 conn(0x7f6868041bb0 legacy=0x7f6868044070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.925+0000 7f688104a700 1 -- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f687c05e230 msgr2=0x7f687c134cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.925+0000 7f688104a700 1 --2- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f687c05e230 0x7f687c134cd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f686c00ec80 tx=0x7f686c006f40).stop 2024-04-26T09:35:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.925+0000 7f688104a700 1 -- 172.21.15.194:0/1879288967 shutdown_connections 2024-04-26T09:35:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.925+0000 7f688104a700 1 --2- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f687c12acb0 0x7f687c1356b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.925+0000 7f688104a700 1 --2- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f687c05e230 0x7f687c134cd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.925+0000 7f688104a700 1 --2- 172.21.15.194:0/1879288967 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f687c065990 0x7f687c1351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:43.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.925+0000 7f688104a700 1 -- 172.21.15.194:0/1879288967 >> 172.21.15.194:0/1879288967 conn(0x7f687c091910 msgr2=0x7f687c12a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:43.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.925+0000 7f688104a700 1 -- 172.21.15.194:0/1879288967 shutdown_connections 2024-04-26T09:35:43.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:43.925+0000 7f688104a700 1 -- 172.21.15.194:0/1879288967 wait complete. 2024-04-26T09:35:43.930 INFO:teuthology.orchestra.run.smithi194.stderr:dumped all 2024-04-26T09:35:43.940 INFO:teuthology.orchestra.run.smithi194.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-26T09:35:42.434502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1188776,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":1188352,"kb_avail":376298584,"statfs":{"total":386547056640,"available":385329750016,"internally_reserved":0,"allocated":401408,"data_stored":98823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1216872448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.476064+0000","last_change":"2024-04-26T09:35:39.476064+0000","last_active":"2024-04-26T09:35:39.476064+0000","last_peered":"2024-04-26T09:35:39.476064+0000","last_clean":"2024-04-26T09:35:39.476064+0000","last_became_active":"2024-04-26T09:35:39.475797+0000","last_became_peered":"2024-04-26T09:35:39.475797+0000","last_unstale":"2024-04-26T09:35:39.476064+0000","last_undegraded":"2024-04-26T09:35:39.476064+0000","last_fullsized":"2024-04-26T09:35:39.476064+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.488994+0000","last_change":"2024-04-26T09:35:39.482000+0000","last_active":"2024-04-26T09:35:39.488994+0000","last_peered":"2024-04-26T09:35:39.488994+0000","last_clean":"2024-04-26T09:35:39.488994+0000","last_became_active":"2024-04-26T09:35:39.481746+0000","last_became_peered":"2024-04-26T09:35:39.481746+0000","last_unstale":"2024-04-26T09:35:39.488994+0000","last_undegraded":"2024-04-26T09:35:39.488994+0000","last_fullsized":"2024-04-26T09:35:39.488994+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.473129+0000","last_change":"2024-04-26T09:35:39.473129+0000","last_active":"2024-04-26T09:35:39.473129+0000","last_peered":"2024-04-26T09:35:39.473129+0000","last_clean":"2024-04-26T09:35:39.473129+0000","last_became_active":"2024-04-26T09:35:39.472876+0000","last_became_peered":"2024-04-26T09:35:39.472876+0000","last_unstale":"2024-04-26T09:35:39.473129+0000","last_undegraded":"2024-04-26T09:35:39.473129+0000","last_fullsized":"2024-04-26T09:35:39.473129+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.482206+0000","last_change":"2024-04-26T09:35:39.482206+0000","last_active":"2024-04-26T09:35:39.482206+0000","last_peered":"2024-04-26T09:35:39.482206+0000","last_clean":"2024-04-26T09:35:39.482206+0000","last_became_active":"2024-04-26T09:35:39.481898+0000","last_became_peered":"2024-04-26T09:35:39.481898+0000","last_unstale":"2024-04-26T09:35:39.482206+0000","last_undegraded":"2024-04-26T09:35:39.482206+0000","last_fullsized":"2024-04-26T09:35:39.482206+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.488973+0000","last_change":"2024-04-26T09:35:39.481874+0000","last_active":"2024-04-26T09:35:39.488973+0000","last_peered":"2024-04-26T09:35:39.488973+0000","last_clean":"2024-04-26T09:35:39.488973+0000","last_became_active":"2024-04-26T09:35:39.481585+0000","last_became_peered":"2024-04-26T09:35:39.481585+0000","last_unstale":"2024-04-26T09:35:39.488973+0000","last_undegraded":"2024-04-26T09:35:39.488973+0000","last_fullsized":"2024-04-26T09:35:39.488973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.478633+0000","last_change":"2024-04-26T09:35:39.478633+0000","last_active":"2024-04-26T09:35:39.478633+0000","last_peered":"2024-04-26T09:35:39.478633+0000","last_clean":"2024-04-26T09:35:39.478633+0000","last_became_active":"2024-04-26T09:35:39.478349+0000","last_became_peered":"2024-04-26T09:35:39.478349+0000","last_unstale":"2024-04-26T09:35:39.478633+0000","last_undegraded":"2024-04-26T09:35:39.478633+0000","last_fullsized":"2024-04-26T09:35:39.478633+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.481494+0000","last_change":"2024-04-26T09:35:39.481494+0000","last_active":"2024-04-26T09:35:39.481494+0000","last_peered":"2024-04-26T09:35:39.481494+0000","last_clean":"2024-04-26T09:35:39.481494+0000","last_became_active":"2024-04-26T09:35:39.481144+0000","last_became_peered":"2024-04-26T09:35:39.481144+0000","last_unstale":"2024-04-26T09:35:39.481494+0000","last_undegraded":"2024-04-26T09:35:39.481494+0000","last_fullsized":"2024-04-26T09:35:39.481494+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.481502+0000","last_change":"2024-04-26T09:35:39.481502+0000","last_active":"2024-04-26T09:35:39.481502+0000","last_peered":"2024-04-26T09:35:39.481502+0000","last_clean":"2024-04-26T09:35:39.481502+0000","last_became_active":"2024-04-26T09:35:39.481116+0000","last_became_peered":"2024-04-26T09:35:39.481116+0000","last_unstale":"2024-04-26T09:35:39.481502+0000","last_undegraded":"2024-04-26T09:35:39.481502+0000","last_fullsized":"2024-04-26T09:35:39.481502+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.488923+0000","last_change":"2024-04-26T09:35:39.481452+0000","last_active":"2024-04-26T09:35:39.488923+0000","last_peered":"2024-04-26T09:35:39.488923+0000","last_clean":"2024-04-26T09:35:39.488923+0000","last_became_active":"2024-04-26T09:35:39.481155+0000","last_became_peered":"2024-04-26T09:35:39.481155+0000","last_unstale":"2024-04-26T09:35:39.488923+0000","last_undegraded":"2024-04-26T09:35:39.488923+0000","last_fullsized":"2024-04-26T09:35:39.488923+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":23768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":23768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074640,"statfs":{"total":96636764160,"available":96332431360,"internally_reserved":0,"allocated":106496,"data_stored":27519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":23768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T09:35:43.941 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T09:35:44.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.051+0000 7f516903a700 1 Processor -- start 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.051+0000 7f516903a700 1 -- start start 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516903a700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f5164128610 0x7f516412aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516903a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f516412aef0 0x7f516412d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516903a700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51640673b0 con 0x7f5164125670 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516903a700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51640697b0 con 0x7f5164128610 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516903a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f5164126160 con 0x7f516412aef0 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516359e700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f516412aef0 0x7f516412d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516259c700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f5164128610 0x7f516412aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f5162d9d700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f5164125670 0x7f5164125a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:51740/0 (socket says 172.21.15.194:51740) 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f5162d9d700 1 -- 172.21.15.194:0/3691803075 learned_addr learned my addr 172.21.15.194:0/3691803075 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516359e700 1 -- 172.21.15.194:0/3691803075 >> v2:172.21.15.194:3300/0 conn(0x7f5164128610 msgr2=0x7f516412aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516359e700 1 --2- 172.21.15.194:0/3691803075 >> v2:172.21.15.194:3300/0 conn(0x7f5164128610 0x7f516412aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516359e700 1 -- 172.21.15.194:0/3691803075 >> v1:172.21.15.194:6789/0 conn(0x7f5164125670 legacy=0x7f5164125a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f5161d9b700 1 -- 172.21.15.194:0/3691803075 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710896405 0 0) 0x7f51640673b0 con 0x7f5164125670 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516359e700 1 -- 172.21.15.194:0/3691803075 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51641252b0 con 0x7f516412aef0 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516259c700 1 --2- 172.21.15.194:0/3691803075 >> v2:172.21.15.194:3300/0 conn(0x7f5164128610 0x7f516412aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f516359e700 1 --2- 172.21.15.194:0/3691803075 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f516412aef0 0x7f516412d2e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f515c009e90 tx=0x7f515c009780).ready entity=mon.2 client_cookie=4318f29c295d4084 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f5161d9b700 1 -- 172.21.15.194:0/3691803075 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c00e070 con 0x7f516412aef0 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f5161d9b700 1 -- 172.21.15.194:0/3691803075 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c00bb20 con 0x7f516412aef0 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.052+0000 7f5161d9b700 1 -- 172.21.15.194:0/3691803075 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c00bea0 con 0x7f516412aef0 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.053+0000 7f516903a700 1 -- 172.21.15.194:0/3691803075 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f516412aef0 msgr2=0x7f516412d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.053+0000 7f516903a700 1 --2- 172.21.15.194:0/3691803075 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f516412aef0 0x7f516412d2e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f515c009e90 tx=0x7f515c009780).stop 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.053+0000 7f516903a700 1 -- 172.21.15.194:0/3691803075 shutdown_connections 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.053+0000 7f516903a700 1 --2- 172.21.15.194:0/3691803075 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f516412aef0 0x7f516412d2e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.053+0000 7f516903a700 1 --2- 172.21.15.194:0/3691803075 >> v2:172.21.15.194:3300/0 conn(0x7f5164128610 0x7f516412aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.053+0000 7f516903a700 1 -- 172.21.15.194:0/3691803075 >> 172.21.15.194:0/3691803075 conn(0x7f51640957b0 msgr2=0x7f5164095bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.053+0000 7f516903a700 1 -- 172.21.15.194:0/3691803075 shutdown_connections 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.053+0000 7f516903a700 1 -- 172.21.15.194:0/3691803075 wait complete. 2024-04-26T09:35:44.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.054+0000 7f516903a700 1 Processor -- start 2024-04-26T09:35:44.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.054+0000 7f516903a700 1 -- start start 2024-04-26T09:35:44.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516903a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5164125670 0x7f5164114ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516903a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5164128610 0x7f51641345e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516903a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f516412aef0 0x7f5164134ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516903a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51641133c0 con 0x7f5164125670 2024-04-26T09:35:44.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516903a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51641156c0 con 0x7f516412aef0 2024-04-26T09:35:44.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516903a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f5164134fc0 con 0x7f5164128610 2024-04-26T09:35:44.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f5162d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5164125670 0x7f5164114ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516259c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5164128610 0x7f51641345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f5162d9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5164125670 0x7f5164114ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48400/0 (socket says 172.21.15.194:48400) 2024-04-26T09:35:44.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516259c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5164128610 0x7f51641345e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:33154/0 (socket says 172.21.15.194:33154) 2024-04-26T09:35:44.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f5162d9d700 1 -- 172.21.15.194:0/1405164858 learned_addr learned my addr 172.21.15.194:0/1405164858 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516359e700 1 --2- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f516412aef0 0x7f5164134ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f5162d9d700 1 -- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5164128610 msgr2=0x7f51641345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f5162d9d700 1 --2- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5164128610 0x7f51641345e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f5162d9d700 1 -- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f516412aef0 msgr2=0x7f5164134ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f5162d9d700 1 --2- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f516412aef0 0x7f5164134ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f5162d9d700 1 -- 172.21.15.194:0/1405164858 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51540092a0 con 0x7f5164125670 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516259c700 1 --2- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5164128610 0x7f51641345e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f5162d9d700 1 --2- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5164125670 0x7f5164114ee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f515800c4c0 tx=0x7f5158007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f514b7fe700 1 -- 172.21.15.194:0/1405164858 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515800ce30 con 0x7f5164125670 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f514b7fe700 1 -- 172.21.15.194:0/1405164858 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51580072e0 con 0x7f5164125670 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516903a700 1 -- 172.21.15.194:0/1405164858 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515c009a20 con 0x7f5164125670 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f514b7fe700 1 -- 172.21.15.194:0/1405164858 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51580057a0 con 0x7f5164125670 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.055+0000 7f516903a700 1 -- 172.21.15.194:0/1405164858 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5164112230 con 0x7f5164125670 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.056+0000 7f514b7fe700 1 -- 172.21.15.194:0/1405164858 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f5158005bd0 con 0x7f5164125670 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.056+0000 7f516903a700 1 -- 172.21.15.194:0/1405164858 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5164092a10 con 0x7f5164125670 2024-04-26T09:35:44.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.056+0000 7f514b7fe700 1 -- 172.21.15.194:0/1405164858 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f5158054920 con 0x7f5164125670 2024-04-26T09:35:44.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.060+0000 7f514b7fe700 1 -- 172.21.15.194:0/1405164858 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f51580354f0 con 0x7f5164125670 2024-04-26T09:35:44.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.196+0000 7f516903a700 1 -- 172.21.15.194:0/1405164858 --> v1:172.21.15.194:6816/72244 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5164063b00 con 0x7f5150041b80 2024-04-26T09:35:44.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.198+0000 7f514b7fe700 1 -- 172.21.15.194:0/1405164858 <== mgr.4111 v1:172.21.15.194:6816/72244 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+32744 (unknown 2965378022 0 3809072774) 0x7f5164063b00 con 0x7f5150041b80 2024-04-26T09:35:44.201 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:44.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.199+0000 7f516903a700 1 -- 172.21.15.194:0/1405164858 >> v1:172.21.15.194:6816/72244 conn(0x7f5150041b80 legacy=0x7f5150044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.199+0000 7f516903a700 1 -- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5164125670 msgr2=0x7f5164114ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.199+0000 7f516903a700 1 --2- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5164125670 0x7f5164114ee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f515800c4c0 tx=0x7f5158007c10).stop 2024-04-26T09:35:44.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.199+0000 7f516903a700 1 -- 172.21.15.194:0/1405164858 shutdown_connections 2024-04-26T09:35:44.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.199+0000 7f516903a700 1 --2- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f516412aef0 0x7f5164134ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.199+0000 7f516903a700 1 --2- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5164125670 0x7f5164114ee0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.199+0000 7f516903a700 1 --2- 172.21.15.194:0/1405164858 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f5164128610 0x7f51641345e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.199+0000 7f516903a700 1 -- 172.21.15.194:0/1405164858 >> 172.21.15.194:0/1405164858 conn(0x7f51640957b0 msgr2=0x7f5164127a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:44.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.199+0000 7f516903a700 1 -- 172.21.15.194:0/1405164858 shutdown_connections 2024-04-26T09:35:44.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.199+0000 7f516903a700 1 -- 172.21.15.194:0/1405164858 wait complete. 2024-04-26T09:35:44.204 INFO:teuthology.orchestra.run.smithi194.stderr:dumped all 2024-04-26T09:35:44.214 INFO:teuthology.orchestra.run.smithi194.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-26T09:35:42.434502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1188776,"kb_used_data":392,"kb_used_omap":0,"kb_used_meta":1188352,"kb_avail":376298584,"statfs":{"total":386547056640,"available":385329750016,"internally_reserved":0,"allocated":401408,"data_stored":98823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1216872448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.476064+0000","last_change":"2024-04-26T09:35:39.476064+0000","last_active":"2024-04-26T09:35:39.476064+0000","last_peered":"2024-04-26T09:35:39.476064+0000","last_clean":"2024-04-26T09:35:39.476064+0000","last_became_active":"2024-04-26T09:35:39.475797+0000","last_became_peered":"2024-04-26T09:35:39.475797+0000","last_unstale":"2024-04-26T09:35:39.476064+0000","last_undegraded":"2024-04-26T09:35:39.476064+0000","last_fullsized":"2024-04-26T09:35:39.476064+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.488994+0000","last_change":"2024-04-26T09:35:39.482000+0000","last_active":"2024-04-26T09:35:39.488994+0000","last_peered":"2024-04-26T09:35:39.488994+0000","last_clean":"2024-04-26T09:35:39.488994+0000","last_became_active":"2024-04-26T09:35:39.481746+0000","last_became_peered":"2024-04-26T09:35:39.481746+0000","last_unstale":"2024-04-26T09:35:39.488994+0000","last_undegraded":"2024-04-26T09:35:39.488994+0000","last_fullsized":"2024-04-26T09:35:39.488994+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.473129+0000","last_change":"2024-04-26T09:35:39.473129+0000","last_active":"2024-04-26T09:35:39.473129+0000","last_peered":"2024-04-26T09:35:39.473129+0000","last_clean":"2024-04-26T09:35:39.473129+0000","last_became_active":"2024-04-26T09:35:39.472876+0000","last_became_peered":"2024-04-26T09:35:39.472876+0000","last_unstale":"2024-04-26T09:35:39.473129+0000","last_undegraded":"2024-04-26T09:35:39.473129+0000","last_fullsized":"2024-04-26T09:35:39.473129+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.482206+0000","last_change":"2024-04-26T09:35:39.482206+0000","last_active":"2024-04-26T09:35:39.482206+0000","last_peered":"2024-04-26T09:35:39.482206+0000","last_clean":"2024-04-26T09:35:39.482206+0000","last_became_active":"2024-04-26T09:35:39.481898+0000","last_became_peered":"2024-04-26T09:35:39.481898+0000","last_unstale":"2024-04-26T09:35:39.482206+0000","last_undegraded":"2024-04-26T09:35:39.482206+0000","last_fullsized":"2024-04-26T09:35:39.482206+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.488973+0000","last_change":"2024-04-26T09:35:39.481874+0000","last_active":"2024-04-26T09:35:39.488973+0000","last_peered":"2024-04-26T09:35:39.488973+0000","last_clean":"2024-04-26T09:35:39.488973+0000","last_became_active":"2024-04-26T09:35:39.481585+0000","last_became_peered":"2024-04-26T09:35:39.481585+0000","last_unstale":"2024-04-26T09:35:39.488973+0000","last_undegraded":"2024-04-26T09:35:39.488973+0000","last_fullsized":"2024-04-26T09:35:39.488973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.478633+0000","last_change":"2024-04-26T09:35:39.478633+0000","last_active":"2024-04-26T09:35:39.478633+0000","last_peered":"2024-04-26T09:35:39.478633+0000","last_clean":"2024-04-26T09:35:39.478633+0000","last_became_active":"2024-04-26T09:35:39.478349+0000","last_became_peered":"2024-04-26T09:35:39.478349+0000","last_unstale":"2024-04-26T09:35:39.478633+0000","last_undegraded":"2024-04-26T09:35:39.478633+0000","last_fullsized":"2024-04-26T09:35:39.478633+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.481494+0000","last_change":"2024-04-26T09:35:39.481494+0000","last_active":"2024-04-26T09:35:39.481494+0000","last_peered":"2024-04-26T09:35:39.481494+0000","last_clean":"2024-04-26T09:35:39.481494+0000","last_became_active":"2024-04-26T09:35:39.481144+0000","last_became_peered":"2024-04-26T09:35:39.481144+0000","last_unstale":"2024-04-26T09:35:39.481494+0000","last_undegraded":"2024-04-26T09:35:39.481494+0000","last_fullsized":"2024-04-26T09:35:39.481494+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.481502+0000","last_change":"2024-04-26T09:35:39.481502+0000","last_active":"2024-04-26T09:35:39.481502+0000","last_peered":"2024-04-26T09:35:39.481502+0000","last_clean":"2024-04-26T09:35:39.481502+0000","last_became_active":"2024-04-26T09:35:39.481116+0000","last_became_peered":"2024-04-26T09:35:39.481116+0000","last_unstale":"2024-04-26T09:35:39.481502+0000","last_undegraded":"2024-04-26T09:35:39.481502+0000","last_fullsized":"2024-04-26T09:35:39.481502+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-26T09:35:39.488923+0000","last_change":"2024-04-26T09:35:39.481452+0000","last_active":"2024-04-26T09:35:39.488923+0000","last_peered":"2024-04-26T09:35:39.488923+0000","last_clean":"2024-04-26T09:35:39.488923+0000","last_became_active":"2024-04-26T09:35:39.481155+0000","last_became_peered":"2024-04-26T09:35:39.481155+0000","last_unstale":"2024-04-26T09:35:39.488923+0000","last_undegraded":"2024-04-26T09:35:39.488923+0000","last_fullsized":"2024-04-26T09:35:39.488923+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":23768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":23768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074640,"statfs":{"total":96636764160,"available":96332431360,"internally_reserved":0,"allocated":106496,"data_stored":27519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30399999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705666,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":23768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T09:35:44.215 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T09:35:44.215 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T09:35:44.215 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-26T09:35:44.215 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-26T09:35:44.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.325+0000 7f1e2dd41700 1 Processor -- start 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.325+0000 7f1e2dd41700 1 -- start start 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.325+0000 7f1e2dd41700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28065f90 0x7f1e28126e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.325+0000 7f1e2dd41700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28127370 0x7f1e28129760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.325+0000 7f1e2dd41700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e280678e0 con 0x7f1e280654a0 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.325+0000 7f1e2dd41700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e28066370 con 0x7f1e28065f90 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.325+0000 7f1e2dd41700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1e28064d90 con 0x7f1e28127370 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.325+0000 7f1e277fe700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f1e280654a0 0x7f1e28065880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:51752/0 (socket says 172.21.15.194:51752) 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.325+0000 7f1e26ffd700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28065f90 0x7f1e28126e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.325+0000 7f1e27fff700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28127370 0x7f1e28129760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.325+0000 7f1e277fe700 1 -- 172.21.15.194:0/1474408643 learned_addr learned my addr 172.21.15.194:0/1474408643 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e267fc700 1 -- 172.21.15.194:0/1474408643 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127085317 0 0) 0x7f1e280678e0 con 0x7f1e280654a0 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e267fc700 1 -- 172.21.15.194:0/1474408643 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e100033f0 con 0x7f1e280654a0 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e27fff700 1 -- 172.21.15.194:0/1474408643 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28065f90 msgr2=0x7f1e28126e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:44.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e27fff700 1 --2- 172.21.15.194:0/1474408643 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28065f90 0x7f1e28126e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e27fff700 1 -- 172.21.15.194:0/1474408643 >> v1:172.21.15.194:6789/0 conn(0x7f1e280654a0 legacy=0x7f1e28065880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e27fff700 1 -- 172.21.15.194:0/1474408643 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e28067620 con 0x7f1e28127370 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e26ffd700 1 --2- 172.21.15.194:0/1474408643 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28065f90 0x7f1e28126e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e27fff700 1 --2- 172.21.15.194:0/1474408643 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28127370 0x7f1e28129760 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f1e2000afd0 tx=0x7f1e2000a890).ready entity=mon.2 client_cookie=fa7ed8e451ab72a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e267fc700 1 -- 172.21.15.194:0/1474408643 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2000f070 con 0x7f1e28127370 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e267fc700 1 -- 172.21.15.194:0/1474408643 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2000a2e0 con 0x7f1e28127370 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e267fc700 1 -- 172.21.15.194:0/1474408643 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2000cb40 con 0x7f1e28127370 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e2dd41700 1 -- 172.21.15.194:0/1474408643 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28127370 msgr2=0x7f1e28129760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e2dd41700 1 --2- 172.21.15.194:0/1474408643 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28127370 0x7f1e28129760 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f1e2000afd0 tx=0x7f1e2000a890).stop 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e2dd41700 1 -- 172.21.15.194:0/1474408643 shutdown_connections 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e2dd41700 1 --2- 172.21.15.194:0/1474408643 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28127370 0x7f1e28129760 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e2dd41700 1 --2- 172.21.15.194:0/1474408643 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28065f90 0x7f1e28126e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e2dd41700 1 -- 172.21.15.194:0/1474408643 >> 172.21.15.194:0/1474408643 conn(0x7f1e28091910 msgr2=0x7f1e28091d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.326+0000 7f1e2dd41700 1 -- 172.21.15.194:0/1474408643 shutdown_connections 2024-04-26T09:35:44.334 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.327+0000 7f1e2dd41700 1 -- 172.21.15.194:0/1474408643 wait complete. 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.327+0000 7f1e2dd41700 1 Processor -- start 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.327+0000 7f1e2dd41700 1 -- start start 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e2dd41700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e280654a0 0x7f1e28134be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e2dd41700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28065f90 0x7f1e281350d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e2dd41700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28127370 0x7f1e281355c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e2dd41700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e28126ea0 con 0x7f1e280654a0 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e2dd41700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e28085b80 con 0x7f1e28127370 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e2dd41700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2808eef0 con 0x7f1e28065f90 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e277fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e280654a0 0x7f1e28134be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e26ffd700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28065f90 0x7f1e281350d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e277fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e280654a0 0x7f1e28134be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48416/0 (socket says 172.21.15.194:48416) 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e27fff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28127370 0x7f1e281355c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e277fe700 1 -- 172.21.15.194:0/349357393 learned_addr learned my addr 172.21.15.194:0/349357393 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e26ffd700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28065f90 0x7f1e281350d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:33166/0 (socket says 172.21.15.194:33166) 2024-04-26T09:35:44.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e26ffd700 1 -- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28127370 msgr2=0x7f1e281355c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e26ffd700 1 --2- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28127370 0x7f1e281355c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e26ffd700 1 -- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e280654a0 msgr2=0x7f1e28134be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e26ffd700 1 --2- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e280654a0 0x7f1e28134be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e26ffd700 1 -- 172.21.15.194:0/349357393 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c0092a0 con 0x7f1e28065f90 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e27fff700 1 --2- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28127370 0x7f1e281355c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e277fe700 1 --2- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e280654a0 0x7f1e28134be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.328+0000 7f1e26ffd700 1 --2- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28065f90 0x7f1e281350d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1e1c00e950 tx=0x7f1e1c009060).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.329+0000 7f1e0ffff700 1 -- 172.21.15.194:0/349357393 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c009580 con 0x7f1e28065f90 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.329+0000 7f1e0ffff700 1 -- 172.21.15.194:0/349357393 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c012600 con 0x7f1e28065f90 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.329+0000 7f1e2dd41700 1 -- 172.21.15.194:0/349357393 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2000ab30 con 0x7f1e28065f90 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.329+0000 7f1e0ffff700 1 -- 172.21.15.194:0/349357393 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1c0129c0 con 0x7f1e28065f90 2024-04-26T09:35:44.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.329+0000 7f1e2dd41700 1 -- 172.21.15.194:0/349357393 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e28084f40 con 0x7f1e28065f90 2024-04-26T09:35:44.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.329+0000 7f1e0ffff700 1 -- 172.21.15.194:0/349357393 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f1e1c012d30 con 0x7f1e28065f90 2024-04-26T09:35:44.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.330+0000 7f1e2dd41700 1 -- 172.21.15.194:0/349357393 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e28122280 con 0x7f1e28065f90 2024-04-26T09:35:44.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.330+0000 7f1e0ffff700 1 -- 172.21.15.194:0/349357393 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f1e1c055540 con 0x7f1e28065f90 2024-04-26T09:35:44.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.333+0000 7f1e0ffff700 1 -- 172.21.15.194:0/349357393 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f1e1c035f40 con 0x7f1e28065f90 2024-04-26T09:35:44.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.496+0000 7f1e2dd41700 1 -- 172.21.15.194:0/349357393 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1e28068180 con 0x7f1e28065f90 2024-04-26T09:35:44.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.496+0000 7f1e0ffff700 1 -- 172.21.15.194:0/349357393 <== mon.2 v2:172.21.15.194:3302/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f1e1c016020 con 0x7f1e28065f90 2024-04-26T09:35:44.500 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:44.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.498+0000 7f1e2dd41700 1 -- 172.21.15.194:0/349357393 >> v1:172.21.15.194:6816/72244 conn(0x7f1e14041b80 legacy=0x7f1e14044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.498+0000 7f1e2dd41700 1 -- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28065f90 msgr2=0x7f1e281350d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.498+0000 7f1e2dd41700 1 --2- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28065f90 0x7f1e281350d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1e1c00e950 tx=0x7f1e1c009060).stop 2024-04-26T09:35:44.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.498+0000 7f1e2dd41700 1 -- 172.21.15.194:0/349357393 shutdown_connections 2024-04-26T09:35:44.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.498+0000 7f1e2dd41700 1 --2- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f1e28127370 0x7f1e281355c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.498+0000 7f1e2dd41700 1 --2- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e280654a0 0x7f1e28134be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.498+0000 7f1e2dd41700 1 --2- 172.21.15.194:0/349357393 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f1e28065f90 0x7f1e281350d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.498+0000 7f1e2dd41700 1 -- 172.21.15.194:0/349357393 >> 172.21.15.194:0/349357393 conn(0x7f1e28091910 msgr2=0x7f1e280928a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:44.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.498+0000 7f1e2dd41700 1 -- 172.21.15.194:0/349357393 shutdown_connections 2024-04-26T09:35:44.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.498+0000 7f1e2dd41700 1 -- 172.21.15.194:0/349357393 wait complete. 2024-04-26T09:35:44.513 INFO:teuthology.orchestra.run.smithi194.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T09:35:44.513 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-26T09:35:44.513 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T09:35:44.524 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T09:35:44.524 INFO:teuthology.task.exec:Running commands on role osd.0 host ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:35:44.524 DEBUG:teuthology.orchestra.run.smithi194:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph osd set-require-min-compat-client pacific' 2024-04-26T09:35:44.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.633+0000 7f01b8c9a700 1 Processor -- start 2024-04-26T09:35:44.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.633+0000 7f01b8c9a700 1 -- start start 2024-04-26T09:35:44.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b8c9a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090700 0x7f01b4126b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b8c9a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090b00 0x7f01b4129090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b8c9a700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b4067380 con 0x7f01b408fc10 2024-04-26T09:35:44.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b8c9a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b4066ef0 con 0x7f01b4090b00 2024-04-26T09:35:44.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b8c9a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f01b40912d0 con 0x7f01b4090700 2024-04-26T09:35:44.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b2d9d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090b00 0x7f01b4129090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b259c700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f01b408fc10 0x7f01b408fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:51764/0 (socket says 172.21.15.194:51764) 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b259c700 1 -- 172.21.15.194:0/3376001225 learned_addr learned my addr 172.21.15.194:0/3376001225 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b1d9b700 1 --2- 172.21.15.194:0/3376001225 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090700 0x7f01b4126b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b159a700 1 -- 172.21.15.194:0/3376001225 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086436323 0 0) 0x7f01b4067380 con 0x7f01b408fc10 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b1d9b700 1 -- 172.21.15.194:0/3376001225 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090b00 msgr2=0x7f01b4129090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b1d9b700 1 --2- 172.21.15.194:0/3376001225 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090b00 0x7f01b4129090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b1d9b700 1 -- 172.21.15.194:0/3376001225 >> v1:172.21.15.194:6789/0 conn(0x7f01b408fc10 legacy=0x7f01b408fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b1d9b700 1 -- 172.21.15.194:0/3376001225 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b4067750 con 0x7f01b4090700 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b2d9d700 1 --2- 172.21.15.194:0/3376001225 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090b00 0x7f01b4129090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b1d9b700 1 --2- 172.21.15.194:0/3376001225 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090700 0x7f01b4126b90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f01a400a930 tx=0x7f01a400a200).ready entity=mon.2 client_cookie=2c2b5a21f3a8246f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b159a700 1 -- 172.21.15.194:0/3376001225 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a400a270 con 0x7f01b4090700 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b159a700 1 -- 172.21.15.194:0/3376001225 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a400c790 con 0x7f01b4090700 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.634+0000 7f01b159a700 1 -- 172.21.15.194:0/3376001225 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a400cb10 con 0x7f01b4090700 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.635+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3376001225 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090700 msgr2=0x7f01b4126b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.635+0000 7f01b8c9a700 1 --2- 172.21.15.194:0/3376001225 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090700 0x7f01b4126b90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f01a400a930 tx=0x7f01a400a200).stop 2024-04-26T09:35:44.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.635+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3376001225 shutdown_connections 2024-04-26T09:35:44.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.635+0000 7f01b8c9a700 1 --2- 172.21.15.194:0/3376001225 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090b00 0x7f01b4129090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.635+0000 7f01b8c9a700 1 --2- 172.21.15.194:0/3376001225 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090700 0x7f01b4126b90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.635+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3376001225 >> 172.21.15.194:0/3376001225 conn(0x7f01b4094db0 msgr2=0x7f01b40951c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:44.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.635+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3376001225 shutdown_connections 2024-04-26T09:35:44.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.635+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3376001225 wait complete. 2024-04-26T09:35:44.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.636+0000 7f01b8c9a700 1 Processor -- start 2024-04-26T09:35:44.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.636+0000 7f01b8c9a700 1 -- start start 2024-04-26T09:35:44.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.636+0000 7f01b8c9a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01b408fc10 0x7f01b4132a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.636+0000 7f01b8c9a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090700 0x7f01b4132f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.636+0000 7f01b8c9a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090b00 0x7f01b4133460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.636+0000 7f01b8c9a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b4133950 con 0x7f01b408fc10 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.636+0000 7f01b8c9a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b40942c0 con 0x7f01b4090700 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.636+0000 7f01b8c9a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f01b4119d80 con 0x7f01b4090b00 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01b408fc10 0x7f01b4132a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b1d9b700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090700 0x7f01b4132f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b259c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01b408fc10 0x7f01b4132a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48418/0 (socket says 172.21.15.194:48418) 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b1d9b700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090700 0x7f01b4132f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56934/0 (socket says 172.21.15.194:56934) 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b259c700 1 -- 172.21.15.194:0/3942224253 learned_addr learned my addr 172.21.15.194:0/3942224253 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b2d9d700 1 --2- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090b00 0x7f01b4133460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b1d9b700 1 -- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090b00 msgr2=0x7f01b4133460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b1d9b700 1 --2- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090b00 0x7f01b4133460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b1d9b700 1 -- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01b408fc10 msgr2=0x7f01b4132a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b1d9b700 1 --2- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01b408fc10 0x7f01b4132a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:44.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b1d9b700 1 -- 172.21.15.194:0/3942224253 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac009a20 con 0x7f01b4090700 2024-04-26T09:35:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b2d9d700 1 --2- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090b00 0x7f01b4133460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b1d9b700 1 --2- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090700 0x7f01b4132f70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f01a4015000 tx=0x7f01a400a200).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01a2ffd700 1 -- 172.21.15.194:0/3942224253 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a4006950 con 0x7f01b4090700 2024-04-26T09:35:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01a2ffd700 1 -- 172.21.15.194:0/3942224253 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a400a270 con 0x7f01b4090700 2024-04-26T09:35:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3942224253 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a400a440 con 0x7f01b4090700 2024-04-26T09:35:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01a2ffd700 1 -- 172.21.15.194:0/3942224253 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a4010410 con 0x7f01b4090700 2024-04-26T09:35:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.637+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3942224253 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b408eb90 con 0x7f01b4090700 2024-04-26T09:35:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.638+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3942224253 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b408f3f0 con 0x7f01b4090700 2024-04-26T09:35:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.638+0000 7f01a2ffd700 1 -- 172.21.15.194:0/3942224253 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f01a4010840 con 0x7f01b4090700 2024-04-26T09:35:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.638+0000 7f01a2ffd700 1 -- 172.21.15.194:0/3942224253 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f01a4056460 con 0x7f01b4090700 2024-04-26T09:35:44.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.642+0000 7f01a2ffd700 1 -- 172.21.15.194:0/3942224253 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f01a403a830 con 0x7f01b4090700 2024-04-26T09:35:44.795 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:44.791+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3942224253 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "pacific"} v 0) v1 -- 0x7f01b4069a30 con 0x7f01b4090700 2024-04-26T09:35:45.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.494+0000 7f01a2ffd700 1 -- 172.21.15.194:0/3942224253 <== mon.1 v2:172.21.15.194:3301/0 7 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "pacific"}]=0 set require_min_compat_client to pacific v13) v1 ==== 143+0+0 (secure 0 0 0) 0x7f01a4014030 con 0x7f01b4090700 2024-04-26T09:35:45.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.496+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3942224253 >> v1:172.21.15.194:6816/72244 conn(0x7f0194041b80 legacy=0x7f0194044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:45.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.496+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090700 msgr2=0x7f01b4132f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:45.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.496+0000 7f01b8c9a700 1 --2- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090700 0x7f01b4132f70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f01a4015000 tx=0x7f01a400a200).stop 2024-04-26T09:35:45.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.497+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3942224253 shutdown_connections 2024-04-26T09:35:45.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.497+0000 7f01b8c9a700 1 --2- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f01b4090b00 0x7f01b4133460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:45.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.497+0000 7f01b8c9a700 1 --2- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01b408fc10 0x7f01b4132a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:45.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.497+0000 7f01b8c9a700 1 --2- 172.21.15.194:0/3942224253 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f01b4090700 0x7f01b4132f70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:45.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.497+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3942224253 >> 172.21.15.194:0/3942224253 conn(0x7f01b4094db0 msgr2=0x7f01b4084fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:45.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.497+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3942224253 shutdown_connections 2024-04-26T09:35:45.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.497+0000 7f01b8c9a700 1 -- 172.21.15.194:0/3942224253 wait complete. 2024-04-26T09:35:45.502 INFO:teuthology.orchestra.run.smithi194.stderr:set require_min_compat_client to pacific 2024-04-26T09:35:45.514 INFO:teuthology.run_tasks:Running task print... 2024-04-26T09:35:45.523 INFO:teuthology.task.print:**** done ceph 2024-04-26T09:35:45.523 INFO:teuthology.run_tasks:Running task print... 2024-04-26T09:35:45.535 INFO:teuthology.task.print:*** upgrading, no cephfs present 2024-04-26T09:35:45.535 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T09:35:45.544 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T09:35:45.544 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:35:45.544 DEBUG:teuthology.orchestra.run.smithi194:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs dump' 2024-04-26T09:35:45.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.657+0000 7fd17e5a3700 1 Processor -- start 2024-04-26T09:35:45.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.657+0000 7fd17e5a3700 1 -- start start 2024-04-26T09:35:45.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.657+0000 7fd17e5a3700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fd178059120 0x7fd178090380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:45.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.657+0000 7fd17e5a3700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd178090cc0 0x7fd17812d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:45.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.657+0000 7fd17e5a3700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd178066d20 con 0x7fd178090870 2024-04-26T09:35:45.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.657+0000 7fd17e5a3700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd17808ef10 con 0x7fd178059120 2024-04-26T09:35:45.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.657+0000 7fd17e5a3700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fd17808ead0 con 0x7fd178090cc0 2024-04-26T09:35:45.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.657+0000 7fd17cb40700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd178090cc0 0x7fd17812d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.657+0000 7fd177fff700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fd178059120 0x7fd178090380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.657+0000 7fd1777fe700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd178090870 0x7fd178129ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:51774/0 (socket says 172.21.15.194:51774) 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.657+0000 7fd1777fe700 1 -- 172.21.15.194:0/3017458466 learned_addr learned my addr 172.21.15.194:0/3017458466 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd177fff700 1 -- 172.21.15.194:0/3017458466 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd178090cc0 msgr2=0x7fd17812d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd177fff700 1 --2- 172.21.15.194:0/3017458466 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd178090cc0 0x7fd17812d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd177fff700 1 -- 172.21.15.194:0/3017458466 >> v1:172.21.15.194:6789/0 conn(0x7fd178090870 legacy=0x7fd178129ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd177fff700 1 -- 172.21.15.194:0/3017458466 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd178091150 con 0x7fd178059120 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd176ffd700 1 -- 172.21.15.194:0/3017458466 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969680878 0 0) 0x7fd178066d20 con 0x7fd178090870 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd17cb40700 1 --2- 172.21.15.194:0/3017458466 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd178090cc0 0x7fd17812d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd177fff700 1 --2- 172.21.15.194:0/3017458466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178059120 0x7fd178090380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fd168009760 tx=0x7fd168009060).ready entity=mon.0 client_cookie=6b72517f8ec183d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd176ffd700 1 -- 172.21.15.194:0/3017458466 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16800f510 con 0x7fd178059120 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd176ffd700 1 -- 172.21.15.194:0/3017458466 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd16800f670 con 0x7fd178059120 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd176ffd700 1 -- 172.21.15.194:0/3017458466 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16800f9f0 con 0x7fd178059120 2024-04-26T09:35:45.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd17e5a3700 1 -- 172.21.15.194:0/3017458466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178059120 msgr2=0x7fd178090380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:45.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd17e5a3700 1 --2- 172.21.15.194:0/3017458466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178059120 0x7fd178090380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fd168009760 tx=0x7fd168009060).stop 2024-04-26T09:35:45.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd17e5a3700 1 -- 172.21.15.194:0/3017458466 shutdown_connections 2024-04-26T09:35:45.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd17e5a3700 1 --2- 172.21.15.194:0/3017458466 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd178090cc0 0x7fd17812d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:45.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd17e5a3700 1 --2- 172.21.15.194:0/3017458466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178059120 0x7fd178090380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:45.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd17e5a3700 1 -- 172.21.15.194:0/3017458466 >> 172.21.15.194:0/3017458466 conn(0x7fd178091a40 msgr2=0x7fd178094990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:45.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd17e5a3700 1 -- 172.21.15.194:0/3017458466 shutdown_connections 2024-04-26T09:35:45.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.658+0000 7fd17e5a3700 1 -- 172.21.15.194:0/3017458466 wait complete. 2024-04-26T09:35:45.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.659+0000 7fd17e5a3700 1 Processor -- start 2024-04-26T09:35:45.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.659+0000 7fd17e5a3700 1 -- start start 2024-04-26T09:35:45.667 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17e5a3700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd178059120 0x7fd17811d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17e5a3700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd178090870 0x7fd178134740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17e5a3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178090cc0 0x7fd178134c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17e5a3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd17811b620 con 0x7fd178090cc0 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17e5a3700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd17811d900 con 0x7fd178059120 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17e5a3700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fd17811da70 con 0x7fd178090870 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17cb40700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178090cc0 0x7fd178134c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd177fff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd178059120 0x7fd17811d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd177fff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd178059120 0x7fd17811d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56936/0 (socket says 172.21.15.194:56936) 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17cb40700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178090cc0 0x7fd178134c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48432/0 (socket says 172.21.15.194:48432) 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd177fff700 1 -- 172.21.15.194:0/4082044449 learned_addr learned my addr 172.21.15.194:0/4082044449 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd1777fe700 1 --2- 172.21.15.194:0/4082044449 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd178090870 0x7fd178134740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:35:45.668 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17cb40700 1 -- 172.21.15.194:0/4082044449 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd178090870 msgr2=0x7fd178134740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17cb40700 1 --2- 172.21.15.194:0/4082044449 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd178090870 0x7fd178134740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17cb40700 1 -- 172.21.15.194:0/4082044449 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd178059120 msgr2=0x7fd17811d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17cb40700 1 --2- 172.21.15.194:0/4082044449 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd178059120 0x7fd17811d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17cb40700 1 -- 172.21.15.194:0/4082044449 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd170009a20 con 0x7fd178090cc0 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd17cb40700 1 --2- 172.21.15.194:0/4082044449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178090cc0 0x7fd178134c30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fd170009f10 tx=0x7fd17000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd174ff9700 1 -- 172.21.15.194:0/4082044449 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd170009c70 con 0x7fd178090cc0 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.660+0000 7fd174ff9700 1 -- 172.21.15.194:0/4082044449 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17000f040 con 0x7fd178090cc0 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.661+0000 7fd174ff9700 1 -- 172.21.15.194:0/4082044449 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17000cd80 con 0x7fd178090cc0 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.661+0000 7fd17e5a3700 1 -- 172.21.15.194:0/4082044449 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1680092a0 con 0x7fd178090cc0 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.661+0000 7fd17e5a3700 1 -- 172.21.15.194:0/4082044449 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd178135570 con 0x7fd178090cc0 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.661+0000 7fd174ff9700 1 -- 172.21.15.194:0/4082044449 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fd170013610 con 0x7fd178090cc0 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.661+0000 7fd17e5a3700 1 -- 172.21.15.194:0/4082044449 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd178047540 con 0x7fd178090cc0 2024-04-26T09:35:45.669 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.662+0000 7fd174ff9700 1 -- 172.21.15.194:0/4082044449 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fd1700559d0 con 0x7fd178090cc0 2024-04-26T09:35:45.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.665+0000 7fd174ff9700 1 -- 172.21.15.194:0/4082044449 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fd170036540 con 0x7fd178090cc0 2024-04-26T09:35:45.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.828+0000 7fd17e5a3700 1 -- 172.21.15.194:0/4082044449 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fd178066170 con 0x7fd178090cc0 2024-04-26T09:35:45.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.828+0000 7fd174ff9700 1 -- 172.21.15.194:0/4082044449 <== mon.0 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 1 v1) v1 ==== 75+0+358 (secure 0 0 0) 0x7fd170039b60 con 0x7fd178090cc0 2024-04-26T09:35:45.832 INFO:teuthology.orchestra.run.smithi194.stdout:e1 2024-04-26T09:35:45.832 INFO:teuthology.orchestra.run.smithi194.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T09:35:45.832 INFO:teuthology.orchestra.run.smithi194.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-26T09:35:45.832 INFO:teuthology.orchestra.run.smithi194.stdout:legacy client fscid: -1 2024-04-26T09:35:45.832 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:35:45.832 INFO:teuthology.orchestra.run.smithi194.stdout:No filesystems configured 2024-04-26T09:35:45.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.829+0000 7fd17e5a3700 1 -- 172.21.15.194:0/4082044449 >> v1:172.21.15.194:6816/72244 conn(0x7fd164041b80 legacy=0x7fd164044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:45.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.829+0000 7fd17e5a3700 1 -- 172.21.15.194:0/4082044449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178090cc0 msgr2=0x7fd178134c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:35:45.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.829+0000 7fd17e5a3700 1 --2- 172.21.15.194:0/4082044449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178090cc0 0x7fd178134c30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fd170009f10 tx=0x7fd17000b040).stop 2024-04-26T09:35:45.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.829+0000 7fd17e5a3700 1 -- 172.21.15.194:0/4082044449 shutdown_connections 2024-04-26T09:35:45.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.829+0000 7fd17e5a3700 1 --2- 172.21.15.194:0/4082044449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd178090cc0 0x7fd178134c30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:45.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.829+0000 7fd17e5a3700 1 --2- 172.21.15.194:0/4082044449 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd178059120 0x7fd17811d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:45.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.829+0000 7fd17e5a3700 1 --2- 172.21.15.194:0/4082044449 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd178090870 0x7fd178134740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:35:45.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.829+0000 7fd17e5a3700 1 -- 172.21.15.194:0/4082044449 >> 172.21.15.194:0/4082044449 conn(0x7fd178091a40 msgr2=0x7fd178093630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:35:45.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.830+0000 7fd17e5a3700 1 -- 172.21.15.194:0/4082044449 shutdown_connections 2024-04-26T09:35:45.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:35:45.830+0000 7fd17e5a3700 1 -- 172.21.15.194:0/4082044449 wait complete. 2024-04-26T09:35:45.834 INFO:teuthology.orchestra.run.smithi194.stderr:dumped fsmap epoch 1 2024-04-26T09:35:45.844 INFO:teuthology.run_tasks:Running task install.upgrade... 2024-04-26T09:35:45.855 INFO:teuthology.task.install:project ceph config {'mon.a': None} overrides {'flavor': 'default', 'sha1': '19b26365dac54ee709df8c5dc2a7b7e89a50aba5'} 2024-04-26T09:35:45.855 INFO:teuthology.task.install:remote ubuntu@smithi194.front.sepia.ceph.com config {'flavor': 'default', 'sha1': '19b26365dac54ee709df8c5dc2a7b7e89a50aba5'} 2024-04-26T09:35:45.855 INFO:teuthology.task.install:extra packages: [] 2024-04-26T09:35:45.862 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T09:35:45.862 INFO:teuthology.task.install:Upgrading ceph rpm packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd 2024-04-26T09:35:45.862 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph-common --qf '%{VERSION}-%{RELEASE}' 2024-04-26T09:35:45.884 INFO:teuthology.orchestra.run.smithi194.stdout:16.2.15-0.el8 2024-04-26T09:35:45.885 INFO:teuthology.packaging:The installed version of ceph-common is 16.2.15-0.el8 2024-04-26T09:35:45.885 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=19b26365dac54ee709df8c5dc2a7b7e89a50aba5 2024-04-26T09:35:46.031 INFO:teuthology.task.install:Ceph rpm upgrade from 16.2.15-0.el8 to 18.2.2-1254.g19b26365 2024-04-26T09:35:46.031 INFO:teuthology.task.install.rpm:Host smithi194 is: centos 8 x86_64 2024-04-26T09:35:46.031 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=19b26365dac54ee709df8c5dc2a7b7e89a50aba5 2024-04-26T09:35:46.175 INFO:teuthology.task.install.rpm:Repo base URL: https://2.chacra.ceph.com/r/ceph/reef/19b26365dac54ee709df8c5dc2a7b7e89a50aba5/centos/8/flavors/default/ 2024-04-26T09:35:46.175 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/yum.repos.d/ceph.repo 2024-04-26T09:35:46.441 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/reef/19b26365dac54ee709df8c5dc2a7b7e89a50aba5/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/19b26365dac54ee709df8c5dc2a7b7e89a50aba5/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/reef/19b26365dac54ee709df8c5dc2a7b7e89a50aba5/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-26T09:35:46.442 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-26T09:35:46.442 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-26T09:35:46.470 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/19b26365dac54ee709df8c5dc2a7b7e89a50aba5/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-26T09:35:46.538 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-26T09:35:46.606 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-26T09:35:46.663 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-04-26T09:35:46.674 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-04-26T09:35:46.935 INFO:teuthology.orchestra.run.smithi194.stdout:71 files removed 2024-04-26T09:35:46.964 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd 2024-04-26T09:35:47.684 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 839 B/s | 257 B 00:00 2024-04-26T09:35:47.976 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-04-26T09:35:48.310 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 4.8 kB/s | 1.5 kB 00:00 2024-04-26T09:40:42.421 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - AppStream 97 kB/s | 28 MB 04:54 2024-04-26T09:40:49.028 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - BaseOS 6.6 MB/s | 10 MB 00:01 2024-04-26T09:40:51.029 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras 46 kB/s | 18 kB 00:00 2024-04-26T09:40:51.355 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras common packages 35 kB/s | 7.7 kB 00:00 2024-04-26T09:40:52.426 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - PowerTools 5.1 MB/s | 5.0 MB 00:00 2024-04-26T09:40:53.849 INFO:teuthology.orchestra.run.smithi194.stdout:Copr repo for python3-asyncssh owned by ceph 7.3 kB/s | 3.5 kB 00:00 2024-04-26T09:40:55.075 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-04-26T09:40:58.934 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 190 kB/s | 24 kB 00:00 2024-04-26T09:41:02.558 INFO:teuthology.orchestra.run.smithi194.stderr:Package cephadm available, but not installed. 2024-04-26T09:41:02.558 INFO:teuthology.orchestra.run.smithi194.stdout:No match for argument: cephadm 2024-04-26T09:41:02.563 INFO:teuthology.orchestra.run.smithi194.stderr:Package ceph-mgr-dashboard available, but not installed. 2024-04-26T09:41:02.564 INFO:teuthology.orchestra.run.smithi194.stdout:No match for argument: ceph-mgr-dashboard 2024-04-26T09:41:02.564 INFO:teuthology.orchestra.run.smithi194.stderr:Package ceph-mgr-diskprediction-local available, but not installed. 2024-04-26T09:41:02.564 INFO:teuthology.orchestra.run.smithi194.stdout:No match for argument: ceph-mgr-diskprediction-local 2024-04-26T09:41:02.565 INFO:teuthology.orchestra.run.smithi194.stderr:Package ceph-mgr-rook available, but not installed. 2024-04-26T09:41:02.565 INFO:teuthology.orchestra.run.smithi194.stdout:No match for argument: ceph-mgr-rook 2024-04-26T09:41:02.565 INFO:teuthology.orchestra.run.smithi194.stderr:Package ceph-mgr-cephadm available, but not installed. 2024-04-26T09:41:02.565 INFO:teuthology.orchestra.run.smithi194.stdout:No match for argument: ceph-mgr-cephadm 2024-04-26T09:41:02.568 INFO:teuthology.orchestra.run.smithi194.stderr:Package ceph-volume available, but not installed. 2024-04-26T09:41:02.568 INFO:teuthology.orchestra.run.smithi194.stdout:No match for argument: ceph-volume 2024-04-26T09:41:02.661 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-26T09:41:02.662 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-04-26T09:41:02.663 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-26T09:41:02.765 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=19b26365dac54ee709df8c5dc2a7b7e89a50aba5 2024-04-26T09:41:02.996 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-26T09:41:03.020 INFO:teuthology.orchestra.run.smithi194.stdout:16.2.15-0.el8 2024-04-26T09:41:03.020 INFO:teuthology.packaging:The installed version of ceph is 16.2.15-0.el8 2024-04-26T09:41:03.021 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 405, in upgrade upgrade_common(ctx, config, upgrade_old_style) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 351, in upgrade_common verify_package_version(ctx, node, remote) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 57, in verify_package_version raise RuntimeError( RuntimeError: ceph version 18.2.2-1254.g19b26365 was not installed, found 16.2.15-0.el8. 2024-04-26T09:41:03.221 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fa69cedd5e4f4ceb9fce36a9f085c19c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 405, in upgrade upgrade_common(ctx, config, upgrade_old_style) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 351, in upgrade_common verify_package_version(ctx, node, remote) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 57, in verify_package_version raise RuntimeError( RuntimeError: ceph version 18.2.2-1254.g19b26365 was not installed, found 16.2.15-0.el8. 2024-04-26T09:41:03.222 DEBUG:teuthology.run_tasks:Unwinding manager install.upgrade 2024-04-26T09:41:03.232 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-26T09:41:03.239 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T09:41:03.240 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T09:41:03.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.356+0000 7fc5336ff700 1 Processor -- start 2024-04-26T09:41:03.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.356+0000 7fc5336ff700 1 -- start start 2024-04-26T09:41:03.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc5336ff700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c065c20 0x7fc52c066000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc5336ff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c0664f0 0x7fc52c121f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc5336ff700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fc52c122480 0x7fc52c122930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc5336ff700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52c0695c0 con 0x7fc52c122480 2024-04-26T09:41:03.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc5336ff700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c069810 con 0x7fc52c0664f0 2024-04-26T09:41:03.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc5336ff700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c122e20 con 0x7fc52c065c20 2024-04-26T09:41:03.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc530c9a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c0664f0 0x7fc52c121f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc531c9c700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fc52c122480 0x7fc52c122930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc531c9c700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fc52c122480 0x7fc52c122930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44406/0 (socket says 172.21.15.194:44406) 2024-04-26T09:41:03.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc530c9a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c0664f0 0x7fc52c121f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56342/0 (socket says 172.21.15.194:56342) 2024-04-26T09:41:03.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc53149b700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c065c20 0x7fc52c066000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc531c9c700 1 -- 172.21.15.194:0/1738728952 learned_addr learned my addr 172.21.15.194:0/1738728952 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:03.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc531c9c700 1 -- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c065c20 msgr2=0x7fc52c066000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:03.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc531c9c700 1 --2- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c065c20 0x7fc52c066000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc531c9c700 1 -- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c0664f0 msgr2=0x7fc52c121f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:03.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc531c9c700 1 --2- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c0664f0 0x7fc52c121f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc531c9c700 1 -- 172.21.15.194:0/1738728952 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c069290 con 0x7fc52c122480 2024-04-26T09:41:03.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc530c9a700 1 --2- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c0664f0 0x7fc52c121f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:03.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc53149b700 1 --2- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c065c20 0x7fc52c066000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:03.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.357+0000 7fc531c9c700 1 --2- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc52c122480 0x7fc52c122930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc52400d1a0 tx=0x7fc52400ba50).ready entity=mon.0 client_cookie=ecd78bd142de227d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:03.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc523fff700 1 -- 172.21.15.194:0/1738728952 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc524010070 con 0x7fc52c122480 2024-04-26T09:41:03.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc523fff700 1 -- 172.21.15.194:0/1738728952 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52400dd60 con 0x7fc52c122480 2024-04-26T09:41:03.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc523fff700 1 -- 172.21.15.194:0/1738728952 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52400b2e0 con 0x7fc52c122480 2024-04-26T09:41:03.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc5336ff700 1 -- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc52c122480 msgr2=0x7fc52c122930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc5336ff700 1 --2- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc52c122480 0x7fc52c122930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fc52400d1a0 tx=0x7fc52400ba50).stop 2024-04-26T09:41:03.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc5336ff700 1 -- 172.21.15.194:0/1738728952 shutdown_connections 2024-04-26T09:41:03.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc5336ff700 1 --2- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc52c122480 0x7fc52c122930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc5336ff700 1 --2- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c065c20 0x7fc52c066000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc5336ff700 1 --2- 172.21.15.194:0/1738728952 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c0664f0 0x7fc52c121f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc5336ff700 1 -- 172.21.15.194:0/1738728952 >> 172.21.15.194:0/1738728952 conn(0x7fc52c0957b0 msgr2=0x7fc52c095bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:03.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc5336ff700 1 -- 172.21.15.194:0/1738728952 shutdown_connections 2024-04-26T09:41:03.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.358+0000 7fc5336ff700 1 -- 172.21.15.194:0/1738728952 wait complete. 2024-04-26T09:41:03.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.359+0000 7fc5336ff700 1 Processor -- start 2024-04-26T09:41:03.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.359+0000 7fc5336ff700 1 -- start start 2024-04-26T09:41:03.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.359+0000 7fc5336ff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c065c20 0x7fc52c1305f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.359+0000 7fc5336ff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc52c0664f0 0x7fc52c130ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.359+0000 7fc5336ff700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c122480 0x7fc52c130fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.359+0000 7fc5336ff700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c113ca0 con 0x7fc52c0664f0 2024-04-26T09:41:03.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.359+0000 7fc5336ff700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c1314c0 con 0x7fc52c065c20 2024-04-26T09:41:03.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.359+0000 7fc5336ff700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c113320 con 0x7fc52c122480 2024-04-26T09:41:03.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc53149b700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c065c20 0x7fc52c1305f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc530c9a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc52c0664f0 0x7fc52c130ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc531c9c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c122480 0x7fc52c130fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc53149b700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c065c20 0x7fc52c1305f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56352/0 (socket says 172.21.15.194:56352) 2024-04-26T09:41:03.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc53149b700 1 -- 172.21.15.194:0/3513767557 learned_addr learned my addr 172.21.15.194:0/3513767557 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:03.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc530c9a700 1 -- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c122480 msgr2=0x7fc52c130fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc530c9a700 1 --2- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c122480 0x7fc52c130fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc530c9a700 1 -- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c065c20 msgr2=0x7fc52c1305f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc530c9a700 1 --2- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c065c20 0x7fc52c1305f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc530c9a700 1 -- 172.21.15.194:0/3513767557 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5280092a0 con 0x7fc52c0664f0 2024-04-26T09:41:03.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc53149b700 1 --2- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c065c20 0x7fc52c1305f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:03.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc531c9c700 1 --2- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c122480 0x7fc52c130fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:03.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc530c9a700 1 --2- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc52c0664f0 0x7fc52c130ae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fc51c00e950 tx=0x7fc51c009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:03.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc521ffb700 1 -- 172.21.15.194:0/3513767557 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c009580 con 0x7fc52c0664f0 2024-04-26T09:41:03.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc521ffb700 1 -- 172.21.15.194:0/3513767557 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc51c012600 con 0x7fc52c0664f0 2024-04-26T09:41:03.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc5336ff700 1 -- 172.21.15.194:0/3513767557 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51c0092a0 con 0x7fc52c0664f0 2024-04-26T09:41:03.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc521ffb700 1 -- 172.21.15.194:0/3513767557 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc51c0129c0 con 0x7fc52c0664f0 2024-04-26T09:41:03.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.360+0000 7fc5336ff700 1 -- 172.21.15.194:0/3513767557 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52400bcf0 con 0x7fc52c0664f0 2024-04-26T09:41:03.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.361+0000 7fc521ffb700 1 -- 172.21.15.194:0/3513767557 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fc51c012d30 con 0x7fc52c0664f0 2024-04-26T09:41:03.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.361+0000 7fc5336ff700 1 -- 172.21.15.194:0/3513767557 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52c092a10 con 0x7fc52c0664f0 2024-04-26T09:41:03.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.362+0000 7fc521ffb700 1 -- 172.21.15.194:0/3513767557 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fc51c055540 con 0x7fc52c0664f0 2024-04-26T09:41:03.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.365+0000 7fc521ffb700 1 -- 172.21.15.194:0/3513767557 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fc51c035f40 con 0x7fc52c0664f0 2024-04-26T09:41:03.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.500+0000 7fc5336ff700 1 -- 172.21.15.194:0/3513767557 --> v1:172.21.15.194:6816/72244 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc52c046ce0 con 0x7fc510041bb0 2024-04-26T09:41:03.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.502+0000 7fc521ffb700 1 -- 172.21.15.194:0/3513767557 <== mgr.4111 v1:172.21.15.194:6816/72244 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+37983 (unknown 2965378022 0 2845037596) 0x7fc52c046ce0 con 0x7fc510041bb0 2024-04-26T09:41:03.504 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:41:03.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.503+0000 7fc5336ff700 1 -- 172.21.15.194:0/3513767557 >> v1:172.21.15.194:6816/72244 conn(0x7fc510041bb0 legacy=0x7fc510044070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.503+0000 7fc5336ff700 1 -- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc52c0664f0 msgr2=0x7fc52c130ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.503+0000 7fc5336ff700 1 --2- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc52c0664f0 0x7fc52c130ae0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fc51c00e950 tx=0x7fc51c009060).stop 2024-04-26T09:41:03.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.503+0000 7fc5336ff700 1 -- 172.21.15.194:0/3513767557 shutdown_connections 2024-04-26T09:41:03.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.503+0000 7fc5336ff700 1 --2- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fc52c122480 0x7fc52c130fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.503+0000 7fc5336ff700 1 --2- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fc52c065c20 0x7fc52c1305f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.503+0000 7fc5336ff700 1 --2- 172.21.15.194:0/3513767557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc52c0664f0 0x7fc52c130ae0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.507 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.503+0000 7fc5336ff700 1 -- 172.21.15.194:0/3513767557 >> 172.21.15.194:0/3513767557 conn(0x7fc52c0957b0 msgr2=0x7fc52c124320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:03.507 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.503+0000 7fc5336ff700 1 -- 172.21.15.194:0/3513767557 shutdown_connections 2024-04-26T09:41:03.507 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.503+0000 7fc5336ff700 1 -- 172.21.15.194:0/3513767557 wait complete. 2024-04-26T09:41:03.507 INFO:teuthology.orchestra.run.smithi194.stderr:dumped all 2024-04-26T09:41:03.516 INFO:teuthology.orchestra.run.smithi194.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-04-26T09:41:02.499015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1188832,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":1188352,"kb_avail":376298528,"statfs":{"total":386547056640,"available":385329692672,"internally_reserved":0,"allocated":458752,"data_stored":122196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1216872448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787045+0000","last_change":"2024-04-26T09:35:39.476064+0000","last_active":"2024-04-26T09:35:42.787045+0000","last_peered":"2024-04-26T09:35:42.787045+0000","last_clean":"2024-04-26T09:35:42.787045+0000","last_became_active":"2024-04-26T09:35:39.475797+0000","last_became_peered":"2024-04-26T09:35:39.475797+0000","last_unstale":"2024-04-26T09:35:42.787045+0000","last_undegraded":"2024-04-26T09:35:42.787045+0000","last_fullsized":"2024-04-26T09:35:42.787045+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.477037+0000","last_change":"2024-04-26T09:35:39.482000+0000","last_active":"2024-04-26T09:35:40.477037+0000","last_peered":"2024-04-26T09:35:40.477037+0000","last_clean":"2024-04-26T09:35:40.477037+0000","last_became_active":"2024-04-26T09:35:39.481746+0000","last_became_peered":"2024-04-26T09:35:39.481746+0000","last_unstale":"2024-04-26T09:35:40.477037+0000","last_undegraded":"2024-04-26T09:35:40.477037+0000","last_fullsized":"2024-04-26T09:35:40.477037+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787082+0000","last_change":"2024-04-26T09:35:39.473129+0000","last_active":"2024-04-26T09:35:42.787082+0000","last_peered":"2024-04-26T09:35:42.787082+0000","last_clean":"2024-04-26T09:35:42.787082+0000","last_became_active":"2024-04-26T09:35:39.472876+0000","last_became_peered":"2024-04-26T09:35:39.472876+0000","last_unstale":"2024-04-26T09:35:42.787082+0000","last_undegraded":"2024-04-26T09:35:42.787082+0000","last_fullsized":"2024-04-26T09:35:42.787082+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787021+0000","last_change":"2024-04-26T09:35:39.482206+0000","last_active":"2024-04-26T09:35:42.787021+0000","last_peered":"2024-04-26T09:35:42.787021+0000","last_clean":"2024-04-26T09:35:42.787021+0000","last_became_active":"2024-04-26T09:35:39.481898+0000","last_became_peered":"2024-04-26T09:35:39.481898+0000","last_unstale":"2024-04-26T09:35:42.787021+0000","last_undegraded":"2024-04-26T09:35:42.787021+0000","last_fullsized":"2024-04-26T09:35:42.787021+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.476890+0000","last_change":"2024-04-26T09:35:39.481874+0000","last_active":"2024-04-26T09:35:40.476890+0000","last_peered":"2024-04-26T09:35:40.476890+0000","last_clean":"2024-04-26T09:35:40.476890+0000","last_became_active":"2024-04-26T09:35:39.481585+0000","last_became_peered":"2024-04-26T09:35:39.481585+0000","last_unstale":"2024-04-26T09:35:40.476890+0000","last_undegraded":"2024-04-26T09:35:40.476890+0000","last_fullsized":"2024-04-26T09:35:40.476890+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787076+0000","last_change":"2024-04-26T09:35:39.478633+0000","last_active":"2024-04-26T09:35:42.787076+0000","last_peered":"2024-04-26T09:35:42.787076+0000","last_clean":"2024-04-26T09:35:42.787076+0000","last_became_active":"2024-04-26T09:35:39.478349+0000","last_became_peered":"2024-04-26T09:35:39.478349+0000","last_unstale":"2024-04-26T09:35:42.787076+0000","last_undegraded":"2024-04-26T09:35:42.787076+0000","last_fullsized":"2024-04-26T09:35:42.787076+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.786959+0000","last_change":"2024-04-26T09:35:39.481494+0000","last_active":"2024-04-26T09:35:42.786959+0000","last_peered":"2024-04-26T09:35:42.786959+0000","last_clean":"2024-04-26T09:35:42.786959+0000","last_became_active":"2024-04-26T09:35:39.481144+0000","last_became_peered":"2024-04-26T09:35:39.481144+0000","last_unstale":"2024-04-26T09:35:42.786959+0000","last_undegraded":"2024-04-26T09:35:42.786959+0000","last_fullsized":"2024-04-26T09:35:42.786959+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787040+0000","last_change":"2024-04-26T09:35:39.481502+0000","last_active":"2024-04-26T09:35:42.787040+0000","last_peered":"2024-04-26T09:35:42.787040+0000","last_clean":"2024-04-26T09:35:42.787040+0000","last_became_active":"2024-04-26T09:35:39.481116+0000","last_became_peered":"2024-04-26T09:35:39.481116+0000","last_unstale":"2024-04-26T09:35:42.787040+0000","last_undegraded":"2024-04-26T09:35:42.787040+0000","last_fullsized":"2024-04-26T09:35:42.787040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.476945+0000","last_change":"2024-04-26T09:35:39.481452+0000","last_active":"2024-04-26T09:35:40.476945+0000","last_peered":"2024-04-26T09:35:40.476945+0000","last_clean":"2024-04-26T09:35:40.476945+0000","last_became_active":"2024-04-26T09:35:39.481155+0000","last_became_peered":"2024-04-26T09:35:39.481155+0000","last_unstale":"2024-04-26T09:35:40.476945+0000","last_undegraded":"2024-04-26T09:35:40.476945+0000","last_fullsized":"2024-04-26T09:35:40.476945+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.34899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.51000000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40300000000000002,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.28499999999999998}]},{"osd":1,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35999999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47799999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.36799999999999999,"15min":0.374},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.23000000000000001}]},{"osd":2,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.40200000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.49299999999999999,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.254},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.57299999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.29899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.22600000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.45900000000000002,"5min":0.498,"15min":0.498},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.51500000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.32200000000000001}]},{"osd":1,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.35399999999999998,"15min":0.35599999999999998},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.45900000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.30399999999999999}]},{"osd":3,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38800000000000001,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48199999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.499,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.27800000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.33800000000000002,"15min":0.33700000000000002},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":0.47199999999999998,"5min":0.504,"15min":0.504},"last":0.437},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34200000000000003,"15min":0.34100000000000003},"min":{"1min":0.186,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47299999999999998,"5min":0.504,"15min":0.504},"last":0.47299999999999998}]},{"osd":2,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.46999999999999997,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.21099999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51900000000000002,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51300000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.51300000000000001}]}]},{"osd":1,"up_from":9,"seq":38654705730,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.39000000000000001},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.308}]},{"osd":2,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"min":{"1min":0.245,"5min":0.185,"15min":0.185},"max":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.35399999999999998},"min":{"1min":0.26000000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.28499999999999998}]},{"osd":3,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.36699999999999999,"15min":0.34399999999999997},"min":{"1min":0.25900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"min":{"1min":0.27600000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55500000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T09:41:03.517 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T09:41:03.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.629+0000 7f987f1ec700 1 Processor -- start 2024-04-26T09:41:03.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.629+0000 7f987f1ec700 1 -- start start 2024-04-26T09:41:03.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f987f1ec700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f98780665b0 0x7f9878066990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f987f1ec700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f987808e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f987f1ec700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98780681f0 con 0x7f9878067280 2024-04-26T09:41:03.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f987f1ec700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98780676d0 con 0x7f98780665b0 2024-04-26T09:41:03.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f987f1ec700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f987806a640 con 0x7f9878066e80 2024-04-26T09:41:03.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f9877fff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f987808e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f987cf88700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f98780665b0 0x7f9878066990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f9877fff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f987808e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56364/0 (socket says 172.21.15.194:56364) 2024-04-26T09:41:03.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f9877fff700 1 -- 172.21.15.194:0/2495619420 learned_addr learned my addr 172.21.15.194:0/2495619420 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:03.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f987d789700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f9878067280 0x7f987808e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:52874/0 (socket says 172.21.15.194:52874) 2024-04-26T09:41:03.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f9877fff700 1 -- 172.21.15.194:0/2495619420 >> v2:172.21.15.194:3300/0 conn(0x7f98780665b0 msgr2=0x7f9878066990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:03.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f9877fff700 1 --2- 172.21.15.194:0/2495619420 >> v2:172.21.15.194:3300/0 conn(0x7f98780665b0 0x7f9878066990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f9877fff700 1 -- 172.21.15.194:0/2495619420 >> v1:172.21.15.194:6789/0 conn(0x7f9878067280 legacy=0x7f987808e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f9877fff700 1 -- 172.21.15.194:0/2495619420 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987805e8c0 con 0x7f9878066e80 2024-04-26T09:41:03.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f98777fe700 1 -- 172.21.15.194:0/2495619420 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339303282 0 0) 0x7f98780681f0 con 0x7f9878067280 2024-04-26T09:41:03.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f987cf88700 1 --2- 172.21.15.194:0/2495619420 >> v2:172.21.15.194:3300/0 conn(0x7f98780665b0 0x7f9878066990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:03.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f9877fff700 1 --2- 172.21.15.194:0/2495619420 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f987808e040 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9864006fd0 tx=0x7f98640090f0).ready entity=mon.1 client_cookie=aa9e5141555a7bfd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:03.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f98777fe700 1 -- 172.21.15.194:0/2495619420 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986400b580 con 0x7f9878066e80 2024-04-26T09:41:03.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f98777fe700 1 -- 172.21.15.194:0/2495619420 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986400b6e0 con 0x7f9878066e80 2024-04-26T09:41:03.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.630+0000 7f98777fe700 1 -- 172.21.15.194:0/2495619420 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986400ba80 con 0x7f9878066e80 2024-04-26T09:41:03.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.631+0000 7f987f1ec700 1 -- 172.21.15.194:0/2495619420 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 msgr2=0x7f987808e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.631+0000 7f987f1ec700 1 --2- 172.21.15.194:0/2495619420 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f987808e040 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9864006fd0 tx=0x7f98640090f0).stop 2024-04-26T09:41:03.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.631+0000 7f987f1ec700 1 -- 172.21.15.194:0/2495619420 shutdown_connections 2024-04-26T09:41:03.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.631+0000 7f987f1ec700 1 --2- 172.21.15.194:0/2495619420 >> v2:172.21.15.194:3300/0 conn(0x7f98780665b0 0x7f9878066990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.631+0000 7f987f1ec700 1 --2- 172.21.15.194:0/2495619420 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f987808e040 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.631+0000 7f987f1ec700 1 -- 172.21.15.194:0/2495619420 >> 172.21.15.194:0/2495619420 conn(0x7f9878091910 msgr2=0x7f9878091d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:03.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.631+0000 7f987f1ec700 1 -- 172.21.15.194:0/2495619420 shutdown_connections 2024-04-26T09:41:03.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.631+0000 7f987f1ec700 1 -- 172.21.15.194:0/2495619420 wait complete. 2024-04-26T09:41:03.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f987f1ec700 1 Processor -- start 2024-04-26T09:41:03.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f987f1ec700 1 -- start start 2024-04-26T09:41:03.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f987f1ec700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98780665b0 0x7f987808e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f987f1ec700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f98780873a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f987f1ec700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9878067280 0x7f9878087890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f987f1ec700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9878122830 con 0x7f98780665b0 2024-04-26T09:41:03.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f987f1ec700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9878089430 con 0x7f9878066e80 2024-04-26T09:41:03.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f987f1ec700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f9878121da0 con 0x7f9878067280 2024-04-26T09:41:03.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f9877fff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f98780873a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f987d789700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9878067280 0x7f9878087890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f9877fff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f98780873a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56370/0 (socket says 172.21.15.194:56370) 2024-04-26T09:41:03.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f9877fff700 1 -- 172.21.15.194:0/1414044944 learned_addr learned my addr 172.21.15.194:0/1414044944 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:03.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.632+0000 7f987cf88700 1 --2- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98780665b0 0x7f987808e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f9877fff700 1 -- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9878067280 msgr2=0x7f9878087890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f9877fff700 1 --2- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9878067280 0x7f9878087890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f9877fff700 1 -- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98780665b0 msgr2=0x7f987808e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f9877fff700 1 --2- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98780665b0 0x7f987808e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f9877fff700 1 -- 172.21.15.194:0/1414044944 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f986c009250 con 0x7f9878066e80 2024-04-26T09:41:03.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f987cf88700 1 --2- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98780665b0 0x7f987808e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:03.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f9877fff700 1 --2- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f98780873a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f9864011000 tx=0x7f9864017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:03.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f98757fa700 1 -- 172.21.15.194:0/1414044944 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986400b580 con 0x7f9878066e80 2024-04-26T09:41:03.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f98757fa700 1 -- 172.21.15.194:0/1414044944 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9864006950 con 0x7f9878066e80 2024-04-26T09:41:03.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f987f1ec700 1 -- 172.21.15.194:0/1414044944 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9864009370 con 0x7f9878066e80 2024-04-26T09:41:03.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f98757fa700 1 -- 172.21.15.194:0/1414044944 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986400be40 con 0x7f9878066e80 2024-04-26T09:41:03.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.633+0000 7f987f1ec700 1 -- 172.21.15.194:0/1414044944 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9878088f00 con 0x7f9878066e80 2024-04-26T09:41:03.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.634+0000 7f98757fa700 1 -- 172.21.15.194:0/1414044944 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f986401c630 con 0x7f9878066e80 2024-04-26T09:41:03.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.634+0000 7f987f1ec700 1 -- 172.21.15.194:0/1414044944 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9878090440 con 0x7f9878066e80 2024-04-26T09:41:03.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.634+0000 7f98757fa700 1 -- 172.21.15.194:0/1414044944 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f9864056d40 con 0x7f9878066e80 2024-04-26T09:41:03.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.638+0000 7f98757fa700 1 -- 172.21.15.194:0/1414044944 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f98640266a0 con 0x7f9878066e80 2024-04-26T09:41:03.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.773+0000 7f987f1ec700 1 -- 172.21.15.194:0/1414044944 --> v1:172.21.15.194:6816/72244 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9878068b60 con 0x7f985c041b80 2024-04-26T09:41:03.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.774+0000 7f98757fa700 1 -- 172.21.15.194:0/1414044944 <== mgr.4111 v1:172.21.15.194:6816/72244 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+37983 (unknown 2965378022 0 2845037596) 0x7f9878068b60 con 0x7f985c041b80 2024-04-26T09:41:03.776 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:41:03.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.776+0000 7f987f1ec700 1 -- 172.21.15.194:0/1414044944 >> v1:172.21.15.194:6816/72244 conn(0x7f985c041b80 legacy=0x7f985c044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.776+0000 7f987f1ec700 1 -- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 msgr2=0x7f98780873a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.776+0000 7f987f1ec700 1 --2- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f98780873a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f9864011000 tx=0x7f9864017000).stop 2024-04-26T09:41:03.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.776+0000 7f987f1ec700 1 -- 172.21.15.194:0/1414044944 shutdown_connections 2024-04-26T09:41:03.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.776+0000 7f987f1ec700 1 --2- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f9878067280 0x7f9878087890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.776+0000 7f987f1ec700 1 --2- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98780665b0 0x7f987808e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.776+0000 7f987f1ec700 1 --2- 172.21.15.194:0/1414044944 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f9878066e80 0x7f98780873a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.776+0000 7f987f1ec700 1 -- 172.21.15.194:0/1414044944 >> 172.21.15.194:0/1414044944 conn(0x7f9878091910 msgr2=0x7f987811bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:03.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.776+0000 7f987f1ec700 1 -- 172.21.15.194:0/1414044944 shutdown_connections 2024-04-26T09:41:03.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.776+0000 7f987f1ec700 1 -- 172.21.15.194:0/1414044944 wait complete. 2024-04-26T09:41:03.779 INFO:teuthology.orchestra.run.smithi194.stderr:dumped all 2024-04-26T09:41:03.789 INFO:teuthology.orchestra.run.smithi194.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-04-26T09:41:02.499015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1188832,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":1188352,"kb_avail":376298528,"statfs":{"total":386547056640,"available":385329692672,"internally_reserved":0,"allocated":458752,"data_stored":122196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1216872448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787045+0000","last_change":"2024-04-26T09:35:39.476064+0000","last_active":"2024-04-26T09:35:42.787045+0000","last_peered":"2024-04-26T09:35:42.787045+0000","last_clean":"2024-04-26T09:35:42.787045+0000","last_became_active":"2024-04-26T09:35:39.475797+0000","last_became_peered":"2024-04-26T09:35:39.475797+0000","last_unstale":"2024-04-26T09:35:42.787045+0000","last_undegraded":"2024-04-26T09:35:42.787045+0000","last_fullsized":"2024-04-26T09:35:42.787045+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.477037+0000","last_change":"2024-04-26T09:35:39.482000+0000","last_active":"2024-04-26T09:35:40.477037+0000","last_peered":"2024-04-26T09:35:40.477037+0000","last_clean":"2024-04-26T09:35:40.477037+0000","last_became_active":"2024-04-26T09:35:39.481746+0000","last_became_peered":"2024-04-26T09:35:39.481746+0000","last_unstale":"2024-04-26T09:35:40.477037+0000","last_undegraded":"2024-04-26T09:35:40.477037+0000","last_fullsized":"2024-04-26T09:35:40.477037+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787082+0000","last_change":"2024-04-26T09:35:39.473129+0000","last_active":"2024-04-26T09:35:42.787082+0000","last_peered":"2024-04-26T09:35:42.787082+0000","last_clean":"2024-04-26T09:35:42.787082+0000","last_became_active":"2024-04-26T09:35:39.472876+0000","last_became_peered":"2024-04-26T09:35:39.472876+0000","last_unstale":"2024-04-26T09:35:42.787082+0000","last_undegraded":"2024-04-26T09:35:42.787082+0000","last_fullsized":"2024-04-26T09:35:42.787082+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787021+0000","last_change":"2024-04-26T09:35:39.482206+0000","last_active":"2024-04-26T09:35:42.787021+0000","last_peered":"2024-04-26T09:35:42.787021+0000","last_clean":"2024-04-26T09:35:42.787021+0000","last_became_active":"2024-04-26T09:35:39.481898+0000","last_became_peered":"2024-04-26T09:35:39.481898+0000","last_unstale":"2024-04-26T09:35:42.787021+0000","last_undegraded":"2024-04-26T09:35:42.787021+0000","last_fullsized":"2024-04-26T09:35:42.787021+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.476890+0000","last_change":"2024-04-26T09:35:39.481874+0000","last_active":"2024-04-26T09:35:40.476890+0000","last_peered":"2024-04-26T09:35:40.476890+0000","last_clean":"2024-04-26T09:35:40.476890+0000","last_became_active":"2024-04-26T09:35:39.481585+0000","last_became_peered":"2024-04-26T09:35:39.481585+0000","last_unstale":"2024-04-26T09:35:40.476890+0000","last_undegraded":"2024-04-26T09:35:40.476890+0000","last_fullsized":"2024-04-26T09:35:40.476890+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787076+0000","last_change":"2024-04-26T09:35:39.478633+0000","last_active":"2024-04-26T09:35:42.787076+0000","last_peered":"2024-04-26T09:35:42.787076+0000","last_clean":"2024-04-26T09:35:42.787076+0000","last_became_active":"2024-04-26T09:35:39.478349+0000","last_became_peered":"2024-04-26T09:35:39.478349+0000","last_unstale":"2024-04-26T09:35:42.787076+0000","last_undegraded":"2024-04-26T09:35:42.787076+0000","last_fullsized":"2024-04-26T09:35:42.787076+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.786959+0000","last_change":"2024-04-26T09:35:39.481494+0000","last_active":"2024-04-26T09:35:42.786959+0000","last_peered":"2024-04-26T09:35:42.786959+0000","last_clean":"2024-04-26T09:35:42.786959+0000","last_became_active":"2024-04-26T09:35:39.481144+0000","last_became_peered":"2024-04-26T09:35:39.481144+0000","last_unstale":"2024-04-26T09:35:42.786959+0000","last_undegraded":"2024-04-26T09:35:42.786959+0000","last_fullsized":"2024-04-26T09:35:42.786959+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787040+0000","last_change":"2024-04-26T09:35:39.481502+0000","last_active":"2024-04-26T09:35:42.787040+0000","last_peered":"2024-04-26T09:35:42.787040+0000","last_clean":"2024-04-26T09:35:42.787040+0000","last_became_active":"2024-04-26T09:35:39.481116+0000","last_became_peered":"2024-04-26T09:35:39.481116+0000","last_unstale":"2024-04-26T09:35:42.787040+0000","last_undegraded":"2024-04-26T09:35:42.787040+0000","last_fullsized":"2024-04-26T09:35:42.787040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.476945+0000","last_change":"2024-04-26T09:35:39.481452+0000","last_active":"2024-04-26T09:35:40.476945+0000","last_peered":"2024-04-26T09:35:40.476945+0000","last_clean":"2024-04-26T09:35:40.476945+0000","last_became_active":"2024-04-26T09:35:39.481155+0000","last_became_peered":"2024-04-26T09:35:39.481155+0000","last_unstale":"2024-04-26T09:35:40.476945+0000","last_undegraded":"2024-04-26T09:35:40.476945+0000","last_fullsized":"2024-04-26T09:35:40.476945+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.34899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.51000000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40300000000000002,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.28499999999999998}]},{"osd":1,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35999999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47799999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.36799999999999999,"15min":0.374},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.23000000000000001}]},{"osd":2,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.40200000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.49299999999999999,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.254},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.57299999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.29899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.22600000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.45900000000000002,"5min":0.498,"15min":0.498},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.51500000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.32200000000000001}]},{"osd":1,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.35399999999999998,"15min":0.35599999999999998},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.45900000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.30399999999999999}]},{"osd":3,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38800000000000001,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48199999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.499,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.27800000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.33800000000000002,"15min":0.33700000000000002},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":0.47199999999999998,"5min":0.504,"15min":0.504},"last":0.437},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34200000000000003,"15min":0.34100000000000003},"min":{"1min":0.186,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47299999999999998,"5min":0.504,"15min":0.504},"last":0.47299999999999998}]},{"osd":2,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.46999999999999997,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.21099999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51900000000000002,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51300000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.51300000000000001}]}]},{"osd":1,"up_from":9,"seq":38654705730,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.39000000000000001},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.308}]},{"osd":2,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"min":{"1min":0.245,"5min":0.185,"15min":0.185},"max":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.35399999999999998},"min":{"1min":0.26000000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.28499999999999998}]},{"osd":3,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.36699999999999999,"15min":0.34399999999999997},"min":{"1min":0.25900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"min":{"1min":0.27600000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55500000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T09:41:03.790 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T09:41:03.790 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T09:41:03.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.902+0000 7fa2edd4a700 1 Processor -- start 2024-04-26T09:41:03.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.902+0000 7fa2edd4a700 1 -- start start 2024-04-26T09:41:03.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2edd4a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa2e8063900 0x7fa2e81232c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2edd4a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e81237b0 0x7fa2e8123bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2edd4a700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e8069d80 con 0x7fa2e8067e30 2024-04-26T09:41:03.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2edd4a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8069a00 con 0x7fa2e8063900 2024-04-26T09:41:03.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2edd4a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8069b90 con 0x7fa2e81237b0 2024-04-26T09:41:03.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2e77fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa2e8063900 0x7fa2e81232c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2e6ffd700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e81237b0 0x7fa2e8123bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2e7fff700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fa2e8067e30 0x7fa2e80682b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:52884/0 (socket says 172.21.15.194:52884) 2024-04-26T09:41:03.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2e7fff700 1 -- 172.21.15.194:0/2197357809 learned_addr learned my addr 172.21.15.194:0/2197357809 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:41:03.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2e67fc700 1 -- 172.21.15.194:0/2197357809 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106748991 0 0) 0x7fa2e8069d80 con 0x7fa2e8067e30 2024-04-26T09:41:03.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2e77fe700 1 -- 172.21.15.194:0/2197357809 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e81237b0 msgr2=0x7fa2e8123bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:03.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2e77fe700 1 --2- 172.21.15.194:0/2197357809 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e81237b0 0x7fa2e8123bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2e77fe700 1 -- 172.21.15.194:0/2197357809 >> v1:172.21.15.194:6789/0 conn(0x7fa2e8067e30 legacy=0x7fa2e80682b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2e77fe700 1 -- 172.21.15.194:0/2197357809 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80695c0 con 0x7fa2e8063900 2024-04-26T09:41:03.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2e6ffd700 1 --2- 172.21.15.194:0/2197357809 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e81237b0 0x7fa2e8123bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:03.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.903+0000 7fa2e77fe700 1 --2- 172.21.15.194:0/2197357809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa2e8063900 0x7fa2e81232c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa2d0009790 tx=0x7fa2d0009060).ready entity=mon.1 client_cookie=efd89f1cae0d5be5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:03.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.904+0000 7fa2e67fc700 1 -- 172.21.15.194:0/2197357809 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d000f510 con 0x7fa2e8063900 2024-04-26T09:41:03.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.904+0000 7fa2e67fc700 1 -- 172.21.15.194:0/2197357809 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d000f670 con 0x7fa2e8063900 2024-04-26T09:41:03.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.904+0000 7fa2e67fc700 1 -- 172.21.15.194:0/2197357809 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d000f9f0 con 0x7fa2e8063900 2024-04-26T09:41:03.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.904+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2197357809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa2e8063900 msgr2=0x7fa2e81232c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.904+0000 7fa2edd4a700 1 --2- 172.21.15.194:0/2197357809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa2e8063900 0x7fa2e81232c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa2d0009790 tx=0x7fa2d0009060).stop 2024-04-26T09:41:03.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.904+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2197357809 shutdown_connections 2024-04-26T09:41:03.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.904+0000 7fa2edd4a700 1 --2- 172.21.15.194:0/2197357809 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa2e8063900 0x7fa2e81232c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.904+0000 7fa2edd4a700 1 --2- 172.21.15.194:0/2197357809 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e81237b0 0x7fa2e8123bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.904+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2197357809 >> 172.21.15.194:0/2197357809 conn(0x7fa2e80957b0 msgr2=0x7fa2e8095bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:03.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.904+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2197357809 shutdown_connections 2024-04-26T09:41:03.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.904+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2197357809 wait complete. 2024-04-26T09:41:03.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.905+0000 7fa2edd4a700 1 Processor -- start 2024-04-26T09:41:03.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.905+0000 7fa2edd4a700 1 -- start start 2024-04-26T09:41:03.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2edd4a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2e8063900 0x7fa2e8119130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2edd4a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e8067e30 0x7fa2e8119620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2edd4a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa2e81237b0 0x7fa2e8119b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:03.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2edd4a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e811a000 con 0x7fa2e8063900 2024-04-26T09:41:03.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2edd4a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8118610 con 0x7fa2e81237b0 2024-04-26T09:41:03.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2edd4a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8121100 con 0x7fa2e8067e30 2024-04-26T09:41:03.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2e6ffd700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e8067e30 0x7fa2e8119620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2e7fff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa2e81237b0 0x7fa2e8119b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2e6ffd700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e8067e30 0x7fa2e8119620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47744/0 (socket says 172.21.15.194:47744) 2024-04-26T09:41:03.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2e6ffd700 1 -- 172.21.15.194:0/2105395797 learned_addr learned my addr 172.21.15.194:0/2105395797 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:03.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2e77fe700 1 --2- 172.21.15.194:0/2105395797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2e8063900 0x7fa2e8119130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:03.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2e6ffd700 1 -- 172.21.15.194:0/2105395797 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa2e81237b0 msgr2=0x7fa2e8119b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2e6ffd700 1 --2- 172.21.15.194:0/2105395797 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa2e81237b0 0x7fa2e8119b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2e6ffd700 1 -- 172.21.15.194:0/2105395797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2e8063900 msgr2=0x7fa2e8119130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:03.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2e6ffd700 1 --2- 172.21.15.194:0/2105395797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2e8063900 0x7fa2e8119130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:03.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2e6ffd700 1 -- 172.21.15.194:0/2105395797 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d80092a0 con 0x7fa2e8067e30 2024-04-26T09:41:03.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2e6ffd700 1 --2- 172.21.15.194:0/2105395797 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e8067e30 0x7fa2e8119620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa2d800ea00 tx=0x7fa2d8009060).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:03.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2dffff700 1 -- 172.21.15.194:0/2105395797 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8009600 con 0x7fa2e8067e30 2024-04-26T09:41:03.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2dffff700 1 -- 172.21.15.194:0/2105395797 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d800c6a0 con 0x7fa2e8067e30 2024-04-26T09:41:03.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2105395797 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d00092a0 con 0x7fa2e8067e30 2024-04-26T09:41:03.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2dffff700 1 -- 172.21.15.194:0/2105395797 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d800ca20 con 0x7fa2e8067e30 2024-04-26T09:41:03.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.906+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2105395797 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e8067730 con 0x7fa2e8067e30 2024-04-26T09:41:03.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.907+0000 7fa2dffff700 1 -- 172.21.15.194:0/2105395797 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fa2d800cd90 con 0x7fa2e8067e30 2024-04-26T09:41:03.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.907+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2105395797 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e8118330 con 0x7fa2e8067e30 2024-04-26T09:41:03.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.907+0000 7fa2dffff700 1 -- 172.21.15.194:0/2105395797 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fa2d80556b0 con 0x7fa2e8067e30 2024-04-26T09:41:03.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:03.911+0000 7fa2dffff700 1 -- 172.21.15.194:0/2105395797 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fa2d8036030 con 0x7fa2e8067e30 2024-04-26T09:41:04.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.047+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2105395797 --> v1:172.21.15.194:6816/72244 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2e8121e80 con 0x7fa2c8041b80 2024-04-26T09:41:04.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.048+0000 7fa2dffff700 1 -- 172.21.15.194:0/2105395797 <== mgr.4111 v1:172.21.15.194:6816/72244 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+37983 (unknown 2965378022 0 2845037596) 0x7fa2e8121e80 con 0x7fa2c8041b80 2024-04-26T09:41:04.051 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:41:04.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.050+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2105395797 >> v1:172.21.15.194:6816/72244 conn(0x7fa2c8041b80 legacy=0x7fa2c8044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:04.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.050+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2105395797 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e8067e30 msgr2=0x7fa2e8119620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:04.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.050+0000 7fa2edd4a700 1 --2- 172.21.15.194:0/2105395797 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e8067e30 0x7fa2e8119620 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa2d800ea00 tx=0x7fa2d8009060).stop 2024-04-26T09:41:04.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.050+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2105395797 shutdown_connections 2024-04-26T09:41:04.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.050+0000 7fa2edd4a700 1 --2- 172.21.15.194:0/2105395797 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fa2e81237b0 0x7fa2e8119b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:04.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.050+0000 7fa2edd4a700 1 --2- 172.21.15.194:0/2105395797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2e8063900 0x7fa2e8119130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:04.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.050+0000 7fa2edd4a700 1 --2- 172.21.15.194:0/2105395797 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fa2e8067e30 0x7fa2e8119620 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:04.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.050+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2105395797 >> 172.21.15.194:0/2105395797 conn(0x7fa2e80957b0 msgr2=0x7fa2e811c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:04.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.050+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2105395797 shutdown_connections 2024-04-26T09:41:04.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.050+0000 7fa2edd4a700 1 -- 172.21.15.194:0/2105395797 wait complete. 2024-04-26T09:41:04.056 INFO:teuthology.orchestra.run.smithi194.stderr:dumped all 2024-04-26T09:41:04.063 INFO:teuthology.orchestra.run.smithi194.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-04-26T09:41:02.499015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1188832,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":1188352,"kb_avail":376298528,"statfs":{"total":386547056640,"available":385329692672,"internally_reserved":0,"allocated":458752,"data_stored":122196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1216872448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787045+0000","last_change":"2024-04-26T09:35:39.476064+0000","last_active":"2024-04-26T09:35:42.787045+0000","last_peered":"2024-04-26T09:35:42.787045+0000","last_clean":"2024-04-26T09:35:42.787045+0000","last_became_active":"2024-04-26T09:35:39.475797+0000","last_became_peered":"2024-04-26T09:35:39.475797+0000","last_unstale":"2024-04-26T09:35:42.787045+0000","last_undegraded":"2024-04-26T09:35:42.787045+0000","last_fullsized":"2024-04-26T09:35:42.787045+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.477037+0000","last_change":"2024-04-26T09:35:39.482000+0000","last_active":"2024-04-26T09:35:40.477037+0000","last_peered":"2024-04-26T09:35:40.477037+0000","last_clean":"2024-04-26T09:35:40.477037+0000","last_became_active":"2024-04-26T09:35:39.481746+0000","last_became_peered":"2024-04-26T09:35:39.481746+0000","last_unstale":"2024-04-26T09:35:40.477037+0000","last_undegraded":"2024-04-26T09:35:40.477037+0000","last_fullsized":"2024-04-26T09:35:40.477037+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787082+0000","last_change":"2024-04-26T09:35:39.473129+0000","last_active":"2024-04-26T09:35:42.787082+0000","last_peered":"2024-04-26T09:35:42.787082+0000","last_clean":"2024-04-26T09:35:42.787082+0000","last_became_active":"2024-04-26T09:35:39.472876+0000","last_became_peered":"2024-04-26T09:35:39.472876+0000","last_unstale":"2024-04-26T09:35:42.787082+0000","last_undegraded":"2024-04-26T09:35:42.787082+0000","last_fullsized":"2024-04-26T09:35:42.787082+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787021+0000","last_change":"2024-04-26T09:35:39.482206+0000","last_active":"2024-04-26T09:35:42.787021+0000","last_peered":"2024-04-26T09:35:42.787021+0000","last_clean":"2024-04-26T09:35:42.787021+0000","last_became_active":"2024-04-26T09:35:39.481898+0000","last_became_peered":"2024-04-26T09:35:39.481898+0000","last_unstale":"2024-04-26T09:35:42.787021+0000","last_undegraded":"2024-04-26T09:35:42.787021+0000","last_fullsized":"2024-04-26T09:35:42.787021+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.476890+0000","last_change":"2024-04-26T09:35:39.481874+0000","last_active":"2024-04-26T09:35:40.476890+0000","last_peered":"2024-04-26T09:35:40.476890+0000","last_clean":"2024-04-26T09:35:40.476890+0000","last_became_active":"2024-04-26T09:35:39.481585+0000","last_became_peered":"2024-04-26T09:35:39.481585+0000","last_unstale":"2024-04-26T09:35:40.476890+0000","last_undegraded":"2024-04-26T09:35:40.476890+0000","last_fullsized":"2024-04-26T09:35:40.476890+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787076+0000","last_change":"2024-04-26T09:35:39.478633+0000","last_active":"2024-04-26T09:35:42.787076+0000","last_peered":"2024-04-26T09:35:42.787076+0000","last_clean":"2024-04-26T09:35:42.787076+0000","last_became_active":"2024-04-26T09:35:39.478349+0000","last_became_peered":"2024-04-26T09:35:39.478349+0000","last_unstale":"2024-04-26T09:35:42.787076+0000","last_undegraded":"2024-04-26T09:35:42.787076+0000","last_fullsized":"2024-04-26T09:35:42.787076+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.786959+0000","last_change":"2024-04-26T09:35:39.481494+0000","last_active":"2024-04-26T09:35:42.786959+0000","last_peered":"2024-04-26T09:35:42.786959+0000","last_clean":"2024-04-26T09:35:42.786959+0000","last_became_active":"2024-04-26T09:35:39.481144+0000","last_became_peered":"2024-04-26T09:35:39.481144+0000","last_unstale":"2024-04-26T09:35:42.786959+0000","last_undegraded":"2024-04-26T09:35:42.786959+0000","last_fullsized":"2024-04-26T09:35:42.786959+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787040+0000","last_change":"2024-04-26T09:35:39.481502+0000","last_active":"2024-04-26T09:35:42.787040+0000","last_peered":"2024-04-26T09:35:42.787040+0000","last_clean":"2024-04-26T09:35:42.787040+0000","last_became_active":"2024-04-26T09:35:39.481116+0000","last_became_peered":"2024-04-26T09:35:39.481116+0000","last_unstale":"2024-04-26T09:35:42.787040+0000","last_undegraded":"2024-04-26T09:35:42.787040+0000","last_fullsized":"2024-04-26T09:35:42.787040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.476945+0000","last_change":"2024-04-26T09:35:39.481452+0000","last_active":"2024-04-26T09:35:40.476945+0000","last_peered":"2024-04-26T09:35:40.476945+0000","last_clean":"2024-04-26T09:35:40.476945+0000","last_became_active":"2024-04-26T09:35:39.481155+0000","last_became_peered":"2024-04-26T09:35:39.481155+0000","last_unstale":"2024-04-26T09:35:40.476945+0000","last_undegraded":"2024-04-26T09:35:40.476945+0000","last_fullsized":"2024-04-26T09:35:40.476945+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.34899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.51000000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.19600000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40300000000000002,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.28499999999999998}]},{"osd":1,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35999999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47799999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.373,"5min":0.36799999999999999,"15min":0.374},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.23000000000000001}]},{"osd":2,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.40200000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.49299999999999999,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.254},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.57299999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.29899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.22600000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.45900000000000002,"5min":0.498,"15min":0.498},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.51500000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.32200000000000001}]},{"osd":1,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.35399999999999998,"15min":0.35599999999999998},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.45900000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.30399999999999999}]},{"osd":3,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38800000000000001,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48199999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.499,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.27800000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705731,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.33800000000000002,"15min":0.33700000000000002},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":0.47199999999999998,"5min":0.504,"15min":0.504},"last":0.437},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34200000000000003,"15min":0.34100000000000003},"min":{"1min":0.186,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47299999999999998,"5min":0.504,"15min":0.504},"last":0.47299999999999998}]},{"osd":2,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.46999999999999997,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.21099999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51900000000000002,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51300000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.51300000000000001}]}]},{"osd":1,"up_from":9,"seq":38654705730,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.39000000000000001},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.308}]},{"osd":2,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"min":{"1min":0.245,"5min":0.185,"15min":0.185},"max":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.35399999999999998},"min":{"1min":0.26000000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.28499999999999998}]},{"osd":3,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.36699999999999999,"15min":0.34399999999999997},"min":{"1min":0.25900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"min":{"1min":0.27600000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55500000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T09:41:04.065 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T09:41:04.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.178+0000 7f4f90a6a700 1 Processor -- start 2024-04-26T09:41:04.181 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.178+0000 7f4f90a6a700 1 -- start start 2024-04-26T09:41:04.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.178+0000 7f4f90a6a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 0x7f4f8c127320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:04.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.178+0000 7f4f90a6a700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f4f8c127810 0x7f4f8c129c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:04.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.178+0000 7f4f90a6a700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c0695c0 con 0x7f4f8c065c20 2024-04-26T09:41:04.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.178+0000 7f4f90a6a700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f8c069810 con 0x7f4f8c127810 2024-04-26T09:41:04.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f90a6a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c069290 con 0x7f4f8c066710 2024-04-26T09:41:04.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f89d9b700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 0x7f4f8c127320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:04.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f89d9b700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 0x7f4f8c127320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56398/0 (socket says 172.21.15.194:56398) 2024-04-26T09:41:04.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f8ad9d700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f4f8c127810 0x7f4f8c129c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:04.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f8a59c700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f4f8c065c20 0x7f4f8c066000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:52890/0 (socket says 172.21.15.194:52890) 2024-04-26T09:41:04.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f89d9b700 1 -- 172.21.15.194:0/2015817597 learned_addr learned my addr 172.21.15.194:0/2015817597 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:04.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f8ad9d700 1 -- 172.21.15.194:0/2015817597 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 msgr2=0x7f4f8c127320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:04.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f8ad9d700 1 --2- 172.21.15.194:0/2015817597 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 0x7f4f8c127320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:04.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f8ad9d700 1 -- 172.21.15.194:0/2015817597 >> v1:172.21.15.194:6789/0 conn(0x7f4f8c065c20 legacy=0x7f4f8c066000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:04.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f8959a700 1 -- 172.21.15.194:0/2015817597 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791393543 0 0) 0x7f4f8c0695c0 con 0x7f4f8c065c20 2024-04-26T09:41:04.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f8ad9d700 1 -- 172.21.15.194:0/2015817597 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c0684c0 con 0x7f4f8c127810 2024-04-26T09:41:04.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f89d9b700 1 --2- 172.21.15.194:0/2015817597 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 0x7f4f8c127320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:04.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f8ad9d700 1 --2- 172.21.15.194:0/2015817597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f8c127810 0x7f4f8c129c00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f4f84009e60 tx=0x7f4f84009780).ready entity=mon.0 client_cookie=3ce9bfbb38af207b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:04.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f8959a700 1 -- 172.21.15.194:0/2015817597 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8400e070 con 0x7f4f8c127810 2024-04-26T09:41:04.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f8959a700 1 -- 172.21.15.194:0/2015817597 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8400bb60 con 0x7f4f8c127810 2024-04-26T09:41:04.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.179+0000 7f4f8959a700 1 -- 172.21.15.194:0/2015817597 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84012410 con 0x7f4f8c127810 2024-04-26T09:41:04.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.180+0000 7f4f90a6a700 1 -- 172.21.15.194:0/2015817597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f8c127810 msgr2=0x7f4f8c129c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:04.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.180+0000 7f4f90a6a700 1 --2- 172.21.15.194:0/2015817597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f8c127810 0x7f4f8c129c00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f4f84009e60 tx=0x7f4f84009780).stop 2024-04-26T09:41:04.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.180+0000 7f4f90a6a700 1 -- 172.21.15.194:0/2015817597 shutdown_connections 2024-04-26T09:41:04.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.180+0000 7f4f90a6a700 1 --2- 172.21.15.194:0/2015817597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f8c127810 0x7f4f8c129c00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:04.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.180+0000 7f4f90a6a700 1 --2- 172.21.15.194:0/2015817597 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 0x7f4f8c127320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:04.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.180+0000 7f4f90a6a700 1 -- 172.21.15.194:0/2015817597 >> 172.21.15.194:0/2015817597 conn(0x7f4f8c0957b0 msgr2=0x7f4f8c095bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:04.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.180+0000 7f4f90a6a700 1 -- 172.21.15.194:0/2015817597 shutdown_connections 2024-04-26T09:41:04.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.180+0000 7f4f90a6a700 1 -- 172.21.15.194:0/2015817597 wait complete. 2024-04-26T09:41:04.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.181+0000 7f4f90a6a700 1 Processor -- start 2024-04-26T09:41:04.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.181+0000 7f4f90a6a700 1 -- start start 2024-04-26T09:41:04.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f90a6a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4f8c065c20 0x7f4f8c10e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:04.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f90a6a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 0x7f4f8c10e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:04.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f90a6a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f8c127810 0x7f4f8c10edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:04.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f90a6a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c111870 con 0x7f4f8c127810 2024-04-26T09:41:04.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f90a6a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c10f740 con 0x7f4f8c066710 2024-04-26T09:41:04.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f90a6a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c111c40 con 0x7f4f8c065c20 2024-04-26T09:41:04.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f8a59c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4f8c065c20 0x7f4f8c10e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:04.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f89d9b700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 0x7f4f8c10e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:04.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f8ad9d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f8c127810 0x7f4f8c10edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:04.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f8a59c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4f8c065c20 0x7f4f8c10e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47756/0 (socket says 172.21.15.194:47756) 2024-04-26T09:41:04.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f8a59c700 1 -- 172.21.15.194:0/1193880868 learned_addr learned my addr 172.21.15.194:0/1193880868 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:04.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f89d9b700 1 -- 172.21.15.194:0/1193880868 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4f8c065c20 msgr2=0x7f4f8c10e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:04.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f89d9b700 1 --2- 172.21.15.194:0/1193880868 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4f8c065c20 0x7f4f8c10e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:04.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f89d9b700 1 -- 172.21.15.194:0/1193880868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f8c127810 msgr2=0x7f4f8c10edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:04.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f89d9b700 1 --2- 172.21.15.194:0/1193880868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f8c127810 0x7f4f8c10edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:04.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f89d9b700 1 -- 172.21.15.194:0/1193880868 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8000b610 con 0x7f4f8c066710 2024-04-26T09:41:04.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f89d9b700 1 --2- 172.21.15.194:0/1193880868 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 0x7f4f8c10e8d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f4f80006fd0 tx=0x7f4f8000b3b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:04.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f7affd700 1 -- 172.21.15.194:0/1193880868 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8000b8f0 con 0x7f4f8c066710 2024-04-26T09:41:04.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f7affd700 1 -- 172.21.15.194:0/1193880868 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f8000ede0 con 0x7f4f8c066710 2024-04-26T09:41:04.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f90a6a700 1 -- 172.21.15.194:0/1193880868 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f84009a20 con 0x7f4f8c066710 2024-04-26T09:41:04.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.182+0000 7f4f7affd700 1 -- 172.21.15.194:0/1193880868 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8001a680 con 0x7f4f8c066710 2024-04-26T09:41:04.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.183+0000 7f4f90a6a700 1 -- 172.21.15.194:0/1193880868 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8c10fed0 con 0x7f4f8c066710 2024-04-26T09:41:04.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.183+0000 7f4f7affd700 1 -- 172.21.15.194:0/1193880868 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f4f80017070 con 0x7f4f8c066710 2024-04-26T09:41:04.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.183+0000 7f4f90a6a700 1 -- 172.21.15.194:0/1193880868 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8c092a10 con 0x7f4f8c066710 2024-04-26T09:41:04.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.184+0000 7f4f7affd700 1 -- 172.21.15.194:0/1193880868 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f4f80057e00 con 0x7f4f8c066710 2024-04-26T09:41:04.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.187+0000 7f4f7affd700 1 -- 172.21.15.194:0/1193880868 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f4f80038800 con 0x7f4f8c066710 2024-04-26T09:41:04.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.331+0000 7f4f90a6a700 1 -- 172.21.15.194:0/1193880868 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f8c046ce0 con 0x7f4f8c066710 2024-04-26T09:41:04.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.331+0000 7f4f7affd700 1 -- 172.21.15.194:0/1193880868 <== mon.1 v2:172.21.15.194:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+8123 (secure 0 0 0) 0x7f4f8003bff0 con 0x7f4f8c066710 2024-04-26T09:41:04.333 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:41:04.334 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":13,"fsid":"be7f6752-916a-4f92-bdb5-9db81c891e5c","created":"2024-04-26T09:35:29.120276+0000","modified":"2024-04-26T09:35:45.490099+0000","last_up_change":"2024-04-26T09:35:37.446610+0000","last_in_change":"2024-04-26T09:35:30.442422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"pacific","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2024-04-26T09:35:37.489778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T09:35:38.288163+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"79a6c3fb-32e2-46b3-81f7-7ab1e3bf3e5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6808","nonce":72865}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6809","nonce":72865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6811","nonce":72865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6810","nonce":72865}]},"public_addr":"172.21.15.194:6808/72865","cluster_addr":"172.21.15.194:6809/72865","heartbeat_back_addr":"172.21.15.194:6811/72865","heartbeat_front_addr":"172.21.15.194:6810/72865","state":["exists","up"]},{"osd":1,"uuid":"be81777d-ab3b-46b3-873e-089f690a2249","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6800","nonce":72864}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6801","nonce":72864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6803","nonce":72864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6802","nonce":72864}]},"public_addr":"172.21.15.194:6800/72864","cluster_addr":"172.21.15.194:6801/72864","heartbeat_back_addr":"172.21.15.194:6803/72864","heartbeat_front_addr":"172.21.15.194:6802/72864","state":["exists","up"]},{"osd":2,"uuid":"efddf30f-dc2a-46d5-81d7-0a5cf1b01735","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6812","nonce":72901}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6813","nonce":72901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6815","nonce":72901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6814","nonce":72901}]},"public_addr":"172.21.15.194:6812/72901","cluster_addr":"172.21.15.194:6813/72901","heartbeat_back_addr":"172.21.15.194:6815/72901","heartbeat_front_addr":"172.21.15.194:6814/72901","state":["exists","up"]},{"osd":3,"uuid":"488b4848-1a54-4d45-b95d-ff5ccc61af12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6804","nonce":72866}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6805","nonce":72866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6807","nonce":72866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.194:6806","nonce":72866}]},"public_addr":"172.21.15.194:6804/72866","cluster_addr":"172.21.15.194:6805/72866","heartbeat_back_addr":"172.21.15.194:6807/72866","heartbeat_front_addr":"172.21.15.194:6806/72866","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.624174+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.581378+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.599989+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T09:35:35.632377+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T09:41:04.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.333+0000 7f4f90a6a700 1 -- 172.21.15.194:0/1193880868 >> v1:172.21.15.194:6816/72244 conn(0x7f4f6c041b80 legacy=0x7f4f6c044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:04.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.333+0000 7f4f90a6a700 1 -- 172.21.15.194:0/1193880868 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 msgr2=0x7f4f8c10e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:04.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.333+0000 7f4f90a6a700 1 --2- 172.21.15.194:0/1193880868 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 0x7f4f8c10e8d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f4f80006fd0 tx=0x7f4f8000b3b0).stop 2024-04-26T09:41:04.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.333+0000 7f4f90a6a700 1 -- 172.21.15.194:0/1193880868 shutdown_connections 2024-04-26T09:41:04.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.333+0000 7f4f90a6a700 1 --2- 172.21.15.194:0/1193880868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f8c127810 0x7f4f8c10edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:04.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.333+0000 7f4f90a6a700 1 --2- 172.21.15.194:0/1193880868 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f4f8c065c20 0x7f4f8c10e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:04.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.333+0000 7f4f90a6a700 1 --2- 172.21.15.194:0/1193880868 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f4f8c066710 0x7f4f8c10e8d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:04.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.333+0000 7f4f90a6a700 1 -- 172.21.15.194:0/1193880868 >> 172.21.15.194:0/1193880868 conn(0x7f4f8c0957b0 msgr2=0x7f4f8c122ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:04.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.333+0000 7f4f90a6a700 1 -- 172.21.15.194:0/1193880868 shutdown_connections 2024-04-26T09:41:04.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:04.333+0000 7f4f90a6a700 1 -- 172.21.15.194:0/1193880868 wait complete. 2024-04-26T09:41:05.347 INFO:tasks.ceph:Scrubbing osd.0 2024-04-26T09:41:05.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-26T09:41:05.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f52a512d700 1 Processor -- start 2024-04-26T09:41:05.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f52a512d700 1 -- start start 2024-04-26T09:41:05.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f52a512d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 0x7f52a0067250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f52a512d700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f52a0067740 0x7f52a011c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f52a512d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f52a011c5c0 0x7f52a011ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f52a512d700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a011d180 con 0x7f52a0067740 2024-04-26T09:41:05.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f52a512d700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0065090 con 0x7f52a011c5c0 2024-04-26T09:41:05.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f52a512d700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f52a011cf60 con 0x7f52a0066e70 2024-04-26T09:41:05.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529ed9d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 0x7f52a0067250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529e59c700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f52a0067740 0x7f52a011c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529f59e700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f52a011c5c0 0x7f52a011ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529ed9d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 0x7f52a0067250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47758/0 (socket says 172.21.15.194:47758) 2024-04-26T09:41:05.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529e59c700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f52a0067740 0x7f52a011c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44468/0 (socket says 172.21.15.194:44468) 2024-04-26T09:41:05.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529ed9d700 1 -- 172.21.15.194:0/2238621474 learned_addr learned my addr 172.21.15.194:0/2238621474 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:05.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529f59e700 1 --2- 172.21.15.194:0/2238621474 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f52a011c5c0 0x7f52a011ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56420/0 (socket says 172.21.15.194:56420) 2024-04-26T09:41:05.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529f59e700 1 -- 172.21.15.194:0/2238621474 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 msgr2=0x7f52a0067250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:05.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529f59e700 1 --2- 172.21.15.194:0/2238621474 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 0x7f52a0067250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529f59e700 1 -- 172.21.15.194:0/2238621474 >> v2:172.21.15.194:3300/0 conn(0x7f52a0067740 msgr2=0x7f52a011c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:05.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529f59e700 1 --2- 172.21.15.194:0/2238621474 >> v2:172.21.15.194:3300/0 conn(0x7f52a0067740 0x7f52a011c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.477+0000 7f529f59e700 1 -- 172.21.15.194:0/2238621474 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a00688f0 con 0x7f52a011c5c0 2024-04-26T09:41:05.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f529ed9d700 1 --2- 172.21.15.194:0/2238621474 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 0x7f52a0067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:05.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f529e59c700 1 --2- 172.21.15.194:0/2238621474 >> v2:172.21.15.194:3300/0 conn(0x7f52a0067740 0x7f52a011c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:05.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f529f59e700 1 --2- 172.21.15.194:0/2238621474 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f52a011c5c0 0x7f52a011ca70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f5298009e90 tx=0x7f5298009780).ready entity=mon.1 client_cookie=5856dccffc1ba556 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:05.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f529dd9b700 1 -- 172.21.15.194:0/2238621474 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529800e070 con 0x7f52a011c5c0 2024-04-26T09:41:05.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f529dd9b700 1 -- 172.21.15.194:0/2238621474 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529800ba70 con 0x7f52a011c5c0 2024-04-26T09:41:05.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f529dd9b700 1 -- 172.21.15.194:0/2238621474 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529800bdf0 con 0x7f52a011c5c0 2024-04-26T09:41:05.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f52a512d700 1 -- 172.21.15.194:0/2238621474 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f52a011c5c0 msgr2=0x7f52a011ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f52a512d700 1 --2- 172.21.15.194:0/2238621474 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f52a011c5c0 0x7f52a011ca70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f5298009e90 tx=0x7f5298009780).stop 2024-04-26T09:41:05.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f52a512d700 1 -- 172.21.15.194:0/2238621474 shutdown_connections 2024-04-26T09:41:05.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f52a512d700 1 --2- 172.21.15.194:0/2238621474 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f52a011c5c0 0x7f52a011ca70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f52a512d700 1 --2- 172.21.15.194:0/2238621474 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 0x7f52a0067250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f52a512d700 1 --2- 172.21.15.194:0/2238621474 >> v2:172.21.15.194:3300/0 conn(0x7f52a0067740 0x7f52a011c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f52a512d700 1 -- 172.21.15.194:0/2238621474 >> 172.21.15.194:0/2238621474 conn(0x7f52a0091bc0 msgr2=0x7f52a0091fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:05.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f52a512d700 1 -- 172.21.15.194:0/2238621474 shutdown_connections 2024-04-26T09:41:05.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.478+0000 7f52a512d700 1 -- 172.21.15.194:0/2238621474 wait complete. 2024-04-26T09:41:05.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f52a512d700 1 Processor -- start 2024-04-26T09:41:05.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f52a512d700 1 -- start start 2024-04-26T09:41:05.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f52a512d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 0x7f52a01353b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f52a512d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52a0067740 0x7f52a01358a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f52a512d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f52a011c5c0 0x7f52a0135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f52a512d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0124ce0 con 0x7f52a0067740 2024-04-26T09:41:05.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f52a512d700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0110080 con 0x7f52a011c5c0 2024-04-26T09:41:05.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f52a512d700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0110620 con 0x7f52a0066e70 2024-04-26T09:41:05.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52a0067740 0x7f52a01358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529ed9d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 0x7f52a01353b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529f59e700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f52a011c5c0 0x7f52a0135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529e59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52a0067740 0x7f52a01358a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44476/0 (socket says 172.21.15.194:44476) 2024-04-26T09:41:05.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529ed9d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 0x7f52a01353b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47770/0 (socket says 172.21.15.194:47770) 2024-04-26T09:41:05.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529e59c700 1 -- 172.21.15.194:0/3914501172 learned_addr learned my addr 172.21.15.194:0/3914501172 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:05.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529ed9d700 1 -- 172.21.15.194:0/3914501172 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f52a011c5c0 msgr2=0x7f52a0135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529ed9d700 1 --2- 172.21.15.194:0/3914501172 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f52a011c5c0 0x7f52a0135d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529ed9d700 1 -- 172.21.15.194:0/3914501172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52a0067740 msgr2=0x7f52a01358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529ed9d700 1 --2- 172.21.15.194:0/3914501172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52a0067740 0x7f52a01358a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529ed9d700 1 -- 172.21.15.194:0/3914501172 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5294009300 con 0x7f52a0066e70 2024-04-26T09:41:05.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.480+0000 7f529e59c700 1 --2- 172.21.15.194:0/3914501172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52a0067740 0x7f52a01358a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:05.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.481+0000 7f529ed9d700 1 --2- 172.21.15.194:0/3914501172 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 0x7f52a01353b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f529400e950 tx=0x7f5294009060).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:05.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.481+0000 7f52877fe700 1 -- 172.21.15.194:0/3914501172 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5294009580 con 0x7f52a0066e70 2024-04-26T09:41:05.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.481+0000 7f52877fe700 1 -- 172.21.15.194:0/3914501172 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5294005880 con 0x7f52a0066e70 2024-04-26T09:41:05.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.481+0000 7f52a512d700 1 -- 172.21.15.194:0/3914501172 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52900092a0 con 0x7f52a0066e70 2024-04-26T09:41:05.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.481+0000 7f52877fe700 1 -- 172.21.15.194:0/3914501172 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5294005c00 con 0x7f52a0066e70 2024-04-26T09:41:05.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.481+0000 7f52a512d700 1 -- 172.21.15.194:0/3914501172 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5298009a20 con 0x7f52a0066e70 2024-04-26T09:41:05.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.482+0000 7f52a512d700 1 -- 172.21.15.194:0/3914501172 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f52a0066680 con 0x7f52a0066e70 2024-04-26T09:41:05.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.482+0000 7f52877fe700 1 -- 172.21.15.194:0/3914501172 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f5294005f70 con 0x7f52a0066e70 2024-04-26T09:41:05.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.482+0000 7f52877fe700 1 -- 172.21.15.194:0/3914501172 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f5294055130 con 0x7f52a0066e70 2024-04-26T09:41:05.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.482+0000 7f52877fe700 1 -- 172.21.15.194:0/3914501172 --> v1:172.21.15.194:6808/72865 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f528c04be30 con 0x7f528c048140 2024-04-26T09:41:05.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.482+0000 7f52877fe700 1 -- 172.21.15.194:0/3914501172 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5294057750 con 0x7f52a0066e70 2024-04-26T09:41:05.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.483+0000 7f52877fe700 1 -- 172.21.15.194:0/3914501172 <== osd.0 v1:172.21.15.194:6808/72865 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f528c04be30 con 0x7f528c048140 2024-04-26T09:41:05.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.496+0000 7f52a512d700 1 -- 172.21.15.194:0/3914501172 --> v1:172.21.15.194:6808/72865 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f52a0062cb0 con 0x7f528c048140 2024-04-26T09:41:05.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.496+0000 7f52877fe700 1 -- 172.21.15.194:0/3914501172 <== osd.0 v1:172.21.15.194:6808/72865 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (unknown 0 0 1509541053) 0x7f529800bbe0 con 0x7f528c048140 2024-04-26T09:41:05.498 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-26T09:41:05.498 INFO:teuthology.orchestra.run.smithi194.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-04-26T09:41:05.498 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-26T09:41:05.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.496+0000 7f52a512d700 1 -- 172.21.15.194:0/3914501172 >> v1:172.21.15.194:6808/72865 conn(0x7f528c048140 legacy=0x7f528c04a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.496+0000 7f52a512d700 1 -- 172.21.15.194:0/3914501172 >> v1:172.21.15.194:6816/72244 conn(0x7f528c041b80 legacy=0x7f528c044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.496+0000 7f52a512d700 1 -- 172.21.15.194:0/3914501172 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 msgr2=0x7f52a01353b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.496+0000 7f52a512d700 1 --2- 172.21.15.194:0/3914501172 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f52a0066e70 0x7f52a01353b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f529400e950 tx=0x7f5294009060).stop 2024-04-26T09:41:05.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.496+0000 7f529f59e700 1 -- 172.21.15.194:0/3914501172 reap_dead start 2024-04-26T09:41:05.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.496+0000 7f52a512d700 1 -- 172.21.15.194:0/3914501172 shutdown_connections 2024-04-26T09:41:05.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.496+0000 7f52a512d700 1 -- 172.21.15.194:0/3914501172 >> 172.21.15.194:0/3914501172 conn(0x7f52a0091bc0 msgr2=0x7f52a0128de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:05.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.496+0000 7f52a512d700 1 -- 172.21.15.194:0/3914501172 shutdown_connections 2024-04-26T09:41:05.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.496+0000 7f52a512d700 1 -- 172.21.15.194:0/3914501172 wait complete. 2024-04-26T09:41:05.509 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-26T09:41:05.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.622+0000 7f25509c3700 1 Processor -- start 2024-04-26T09:41:05.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.623+0000 7f25509c3700 1 -- start start 2024-04-26T09:41:05.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.623+0000 7f25509c3700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f254c0903d0 0x7f254c0907b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.623+0000 7f25509c3700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c067e10 0x7f254c090ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.623+0000 7f25509c3700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254c0673b0 con 0x7f254c08e040 2024-04-26T09:41:05.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.623+0000 7f25509c3700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254c067a60 con 0x7f254c0903d0 2024-04-26T09:41:05.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.623+0000 7f25509c3700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c0682c0 con 0x7f254c067e10 2024-04-26T09:41:05.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.623+0000 7f254a59c700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f254c0903d0 0x7f254c0907b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.623+0000 7f254ad9d700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f254c08e040 0x7f254c08e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:52906/0 (socket says 172.21.15.194:52906) 2024-04-26T09:41:05.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.623+0000 7f254ad9d700 1 -- 172.21.15.194:0/913415882 learned_addr learned my addr 172.21.15.194:0/913415882 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:41:05.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.623+0000 7f2549d9b700 1 --2- 172.21.15.194:0/913415882 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c067e10 0x7f254c090ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.624+0000 7f254a59c700 1 -- 172.21.15.194:0/913415882 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c067e10 msgr2=0x7f254c090ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:05.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.624+0000 7f254a59c700 1 --2- 172.21.15.194:0/913415882 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c067e10 0x7f254c090ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.624+0000 7f254a59c700 1 -- 172.21.15.194:0/913415882 >> v1:172.21.15.194:6789/0 conn(0x7f254c08e040 legacy=0x7f254c08e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.624+0000 7f254a59c700 1 -- 172.21.15.194:0/913415882 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c091190 con 0x7f254c0903d0 2024-04-26T09:41:05.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.624+0000 7f254959a700 1 -- 172.21.15.194:0/913415882 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677242254 0 0) 0x7f254c0673b0 con 0x7f254c08e040 2024-04-26T09:41:05.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.624+0000 7f2549d9b700 1 --2- 172.21.15.194:0/913415882 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c067e10 0x7f254c090ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:05.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.624+0000 7f254a59c700 1 --2- 172.21.15.194:0/913415882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f254c0903d0 0x7f254c0907b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f2540006fd0 tx=0x7f2540006f40).ready entity=mon.0 client_cookie=eff0970678d7c3a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:05.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.624+0000 7f254959a700 1 -- 172.21.15.194:0/913415882 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2540005880 con 0x7f254c0903d0 2024-04-26T09:41:05.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.624+0000 7f254959a700 1 -- 172.21.15.194:0/913415882 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25400059e0 con 0x7f254c0903d0 2024-04-26T09:41:05.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.624+0000 7f254959a700 1 -- 172.21.15.194:0/913415882 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2540005d60 con 0x7f254c0903d0 2024-04-26T09:41:05.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.625+0000 7f25509c3700 1 -- 172.21.15.194:0/913415882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f254c0903d0 msgr2=0x7f254c0907b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.625+0000 7f25509c3700 1 --2- 172.21.15.194:0/913415882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f254c0903d0 0x7f254c0907b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f2540006fd0 tx=0x7f2540006f40).stop 2024-04-26T09:41:05.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.625+0000 7f25509c3700 1 -- 172.21.15.194:0/913415882 shutdown_connections 2024-04-26T09:41:05.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.625+0000 7f25509c3700 1 --2- 172.21.15.194:0/913415882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f254c0903d0 0x7f254c0907b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.625+0000 7f25509c3700 1 --2- 172.21.15.194:0/913415882 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c067e10 0x7f254c090ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.625+0000 7f25509c3700 1 -- 172.21.15.194:0/913415882 >> 172.21.15.194:0/913415882 conn(0x7f254c0914c0 msgr2=0x7f254c0918d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:05.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.625+0000 7f25509c3700 1 -- 172.21.15.194:0/913415882 shutdown_connections 2024-04-26T09:41:05.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.625+0000 7f25509c3700 1 -- 172.21.15.194:0/913415882 wait complete. 2024-04-26T09:41:05.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f25509c3700 1 Processor -- start 2024-04-26T09:41:05.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f25509c3700 1 -- start start 2024-04-26T09:41:05.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f25509c3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f254c067e10 0x7f254c0870e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f25509c3700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c08e040 0x7f254c0875d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f25509c3700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f254c0903d0 0x7f254c08b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f25509c3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254c084d10 con 0x7f254c067e10 2024-04-26T09:41:05.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f25509c3700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c0848a0 con 0x7f254c08e040 2024-04-26T09:41:05.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f25509c3700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f254c0844d0 con 0x7f254c0903d0 2024-04-26T09:41:05.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f254a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f254c067e10 0x7f254c0870e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f254a59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f254c067e10 0x7f254c0870e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44496/0 (socket says 172.21.15.194:44496) 2024-04-26T09:41:05.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f254a59c700 1 -- 172.21.15.194:0/2755304238 learned_addr learned my addr 172.21.15.194:0/2755304238 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:05.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f254ad9d700 1 --2- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f254c0903d0 0x7f254c08b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f2549d9b700 1 --2- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c08e040 0x7f254c0875d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f2549d9b700 1 -- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f254c0903d0 msgr2=0x7f254c08b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f2549d9b700 1 --2- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f254c0903d0 0x7f254c08b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f2549d9b700 1 -- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f254c067e10 msgr2=0x7f254c0870e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.626+0000 7f2549d9b700 1 --2- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f254c067e10 0x7f254c0870e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.627+0000 7f2549d9b700 1 -- 172.21.15.194:0/2755304238 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c0092a0 con 0x7f254c08e040 2024-04-26T09:41:05.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.627+0000 7f254a59c700 1 --2- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f254c067e10 0x7f254c0870e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:05.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.627+0000 7f254ad9d700 1 --2- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f254c0903d0 0x7f254c08b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:05.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.627+0000 7f2549d9b700 1 --2- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c08e040 0x7f254c0875d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f254c06a840 tx=0x7f253c009060).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:05.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.627+0000 7f253affd700 1 -- 172.21.15.194:0/2755304238 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c009580 con 0x7f254c08e040 2024-04-26T09:41:05.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.627+0000 7f253affd700 1 -- 172.21.15.194:0/2755304238 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253c012600 con 0x7f254c08e040 2024-04-26T09:41:05.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.627+0000 7f25509c3700 1 -- 172.21.15.194:0/2755304238 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2540009250 con 0x7f254c08e040 2024-04-26T09:41:05.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.627+0000 7f253affd700 1 -- 172.21.15.194:0/2755304238 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c0129a0 con 0x7f254c08e040 2024-04-26T09:41:05.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.627+0000 7f25509c3700 1 -- 172.21.15.194:0/2755304238 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254c087c80 con 0x7f254c08e040 2024-04-26T09:41:05.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.628+0000 7f253affd700 1 -- 172.21.15.194:0/2755304238 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f253c012d10 con 0x7f254c08e040 2024-04-26T09:41:05.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.628+0000 7f253affd700 1 -- 172.21.15.194:0/2755304238 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f253c055500 con 0x7f254c08e040 2024-04-26T09:41:05.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.628+0000 7f25509c3700 1 -- 172.21.15.194:0/2755304238 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254c08ce70 con 0x7f254c08e040 2024-04-26T09:41:05.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.632+0000 7f253affd700 1 -- 172.21.15.194:0/2755304238 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f253c0360d0 con 0x7f254c08e040 2024-04-26T09:41:05.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.772+0000 7f25509c3700 1 -- 172.21.15.194:0/2755304238 --> v1:172.21.15.194:6816/72244 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f254c05e450 con 0x7f252c041b80 2024-04-26T09:41:05.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.772+0000 7f253affd700 1 -- 172.21.15.194:0/2755304238 <== mgr.4111 v1:172.21.15.194:6816/72244 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (unknown 2057319476 0 0) 0x7f254c05e450 con 0x7f252c041b80 2024-04-26T09:41:05.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.774+0000 7f25509c3700 1 -- 172.21.15.194:0/2755304238 >> v1:172.21.15.194:6816/72244 conn(0x7f252c041b80 legacy=0x7f252c044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.774+0000 7f25509c3700 1 -- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c08e040 msgr2=0x7f254c0875d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.774+0000 7f25509c3700 1 --2- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c08e040 0x7f254c0875d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f254c06a840 tx=0x7f253c009060).stop 2024-04-26T09:41:05.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.774+0000 7f25509c3700 1 -- 172.21.15.194:0/2755304238 shutdown_connections 2024-04-26T09:41:05.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.774+0000 7f25509c3700 1 --2- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f254c0903d0 0x7f254c08b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.774+0000 7f25509c3700 1 --2- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f254c067e10 0x7f254c0870e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.774+0000 7f25509c3700 1 --2- 172.21.15.194:0/2755304238 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f254c08e040 0x7f254c0875d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.774+0000 7f25509c3700 1 -- 172.21.15.194:0/2755304238 >> 172.21.15.194:0/2755304238 conn(0x7f254c0914c0 msgr2=0x7f254c08d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:05.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.774+0000 7f25509c3700 1 -- 172.21.15.194:0/2755304238 shutdown_connections 2024-04-26T09:41:05.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.774+0000 7f25509c3700 1 -- 172.21.15.194:0/2755304238 wait complete. 2024-04-26T09:41:05.777 INFO:teuthology.orchestra.run.smithi194.stderr:instructed osd(s) 0 to deep-scrub 2024-04-26T09:41:05.787 INFO:tasks.ceph:Scrubbing osd.1 2024-04-26T09:41:05.787 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-26T09:41:05.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.903+0000 7efceb9a0700 1 Processor -- start 2024-04-26T09:41:05.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.903+0000 7efceb9a0700 1 -- start start 2024-04-26T09:41:05.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efceb9a0700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7efce40904a0 0x7efce4090880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efceb9a0700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce408e750 0x7efce412a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efceb9a0700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce40676b0 con 0x7efce4090d70 2024-04-26T09:41:05.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efceb9a0700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce40682b0 con 0x7efce40904a0 2024-04-26T09:41:05.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efceb9a0700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce405e450 con 0x7efce408e750 2024-04-26T09:41:05.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efce9f3d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce408e750 0x7efce412a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efce973c700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7efce40904a0 0x7efce4090880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efce8f3b700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7efce4090d70 0x7efce408e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:52908/0 (socket says 172.21.15.194:52908) 2024-04-26T09:41:05.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efce9f3d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce408e750 0x7efce412a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56456/0 (socket says 172.21.15.194:56456) 2024-04-26T09:41:05.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efce8f3b700 1 -- 172.21.15.194:0/3411339931 learned_addr learned my addr 172.21.15.194:0/3411339931 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:41:05.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efce9f3d700 1 -- 172.21.15.194:0/3411339931 >> v2:172.21.15.194:3300/0 conn(0x7efce40904a0 msgr2=0x7efce4090880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:05.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efce9f3d700 1 --2- 172.21.15.194:0/3411339931 >> v2:172.21.15.194:3300/0 conn(0x7efce40904a0 0x7efce4090880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efce9f3d700 1 -- 172.21.15.194:0/3411339931 >> v1:172.21.15.194:6789/0 conn(0x7efce4090d70 legacy=0x7efce408e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efce9f3d700 1 -- 172.21.15.194:0/3411339931 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce40672c0 con 0x7efce408e750 2024-04-26T09:41:05.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efcdbfff700 1 -- 172.21.15.194:0/3411339931 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312200699 0 0) 0x7efce40676b0 con 0x7efce4090d70 2024-04-26T09:41:05.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efce973c700 1 --2- 172.21.15.194:0/3411339931 >> v2:172.21.15.194:3300/0 conn(0x7efce40904a0 0x7efce4090880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:05.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.904+0000 7efce9f3d700 1 --2- 172.21.15.194:0/3411339931 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce408e750 0x7efce412a4b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7efcdc009e60 tx=0x7efcdc009780).ready entity=mon.1 client_cookie=2f1942928a5ee975 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:05.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.905+0000 7efcdbfff700 1 -- 172.21.15.194:0/3411339931 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc00e070 con 0x7efce408e750 2024-04-26T09:41:05.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.905+0000 7efcdbfff700 1 -- 172.21.15.194:0/3411339931 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc00bb60 con 0x7efce408e750 2024-04-26T09:41:05.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.905+0000 7efcdbfff700 1 -- 172.21.15.194:0/3411339931 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc012410 con 0x7efce408e750 2024-04-26T09:41:05.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.905+0000 7efceb9a0700 1 -- 172.21.15.194:0/3411339931 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce408e750 msgr2=0x7efce412a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.905+0000 7efceb9a0700 1 --2- 172.21.15.194:0/3411339931 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce408e750 0x7efce412a4b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7efcdc009e60 tx=0x7efcdc009780).stop 2024-04-26T09:41:05.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.905+0000 7efceb9a0700 1 -- 172.21.15.194:0/3411339931 shutdown_connections 2024-04-26T09:41:05.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.905+0000 7efceb9a0700 1 --2- 172.21.15.194:0/3411339931 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce408e750 0x7efce412a4b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.905+0000 7efceb9a0700 1 --2- 172.21.15.194:0/3411339931 >> v2:172.21.15.194:3300/0 conn(0x7efce40904a0 0x7efce4090880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.905+0000 7efceb9a0700 1 -- 172.21.15.194:0/3411339931 >> 172.21.15.194:0/3411339931 conn(0x7efce4094a70 msgr2=0x7efce4094e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:05.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.905+0000 7efceb9a0700 1 -- 172.21.15.194:0/3411339931 shutdown_connections 2024-04-26T09:41:05.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.905+0000 7efceb9a0700 1 -- 172.21.15.194:0/3411339931 wait complete. 2024-04-26T09:41:05.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efceb9a0700 1 Processor -- start 2024-04-26T09:41:05.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efceb9a0700 1 -- start start 2024-04-26T09:41:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efceb9a0700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efce408e750 0x7efce408baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efceb9a0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efce40904a0 0x7efce408bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efceb9a0700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce4090d70 0x7efce4134670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efceb9a0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce4132100 con 0x7efce40904a0 2024-04-26T09:41:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efceb9a0700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce408c990 con 0x7efce4090d70 2024-04-26T09:41:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efceb9a0700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7efce411dc00 con 0x7efce408e750 2024-04-26T09:41:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efce973c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efce408e750 0x7efce408baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efce8f3b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efce40904a0 0x7efce408bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efce973c700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efce408e750 0x7efce408baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47790/0 (socket says 172.21.15.194:47790) 2024-04-26T09:41:05.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efce973c700 1 -- 172.21.15.194:0/675143639 learned_addr learned my addr 172.21.15.194:0/675143639 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:05.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.906+0000 7efce9f3d700 1 --2- 172.21.15.194:0/675143639 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce4090d70 0x7efce4134670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:05.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efce973c700 1 -- 172.21.15.194:0/675143639 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce4090d70 msgr2=0x7efce4134670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efce973c700 1 --2- 172.21.15.194:0/675143639 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce4090d70 0x7efce4134670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efce973c700 1 -- 172.21.15.194:0/675143639 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efce40904a0 msgr2=0x7efce408bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efce973c700 1 --2- 172.21.15.194:0/675143639 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efce40904a0 0x7efce408bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:05.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efce973c700 1 -- 172.21.15.194:0/675143639 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd4009250 con 0x7efce408e750 2024-04-26T09:41:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efce9f3d700 1 --2- 172.21.15.194:0/675143639 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7efce4090d70 0x7efce4134670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efce8f3b700 1 --2- 172.21.15.194:0/675143639 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efce40904a0 0x7efce408bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efce973c700 1 --2- 172.21.15.194:0/675143639 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efce408e750 0x7efce408baa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7efcd400ec80 tx=0x7efcd4006f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efcd9ffb700 1 -- 172.21.15.194:0/675143639 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4005880 con 0x7efce408e750 2024-04-26T09:41:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efcd9ffb700 1 -- 172.21.15.194:0/675143639 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd40059e0 con 0x7efce408e750 2024-04-26T09:41:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efceb9a0700 1 -- 172.21.15.194:0/675143639 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcdc009a20 con 0x7efce408e750 2024-04-26T09:41:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efcd9ffb700 1 -- 172.21.15.194:0/675143639 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd4005d60 con 0x7efce408e750 2024-04-26T09:41:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.907+0000 7efceb9a0700 1 -- 172.21.15.194:0/675143639 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce408c6c0 con 0x7efce408e750 2024-04-26T09:41:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.908+0000 7efceb9a0700 1 -- 172.21.15.194:0/675143639 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efce40923d0 con 0x7efce408e750 2024-04-26T09:41:05.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.908+0000 7efcd9ffb700 1 -- 172.21.15.194:0/675143639 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7efcd400ca60 con 0x7efce408e750 2024-04-26T09:41:05.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.908+0000 7efcd9ffb700 1 -- 172.21.15.194:0/675143639 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7efcd4055380 con 0x7efce408e750 2024-04-26T09:41:05.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.908+0000 7efcd9ffb700 1 -- 172.21.15.194:0/675143639 --> v1:172.21.15.194:6800/72864 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efcc804bfa0 con 0x7efcc80482b0 2024-04-26T09:41:05.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.908+0000 7efcd9ffb700 1 -- 172.21.15.194:0/675143639 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7efcd40579a0 con 0x7efce408e750 2024-04-26T09:41:05.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.909+0000 7efcd9ffb700 1 -- 172.21.15.194:0/675143639 <== osd.1 v1:172.21.15.194:6800/72864 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7efcc804bfa0 con 0x7efcc80482b0 2024-04-26T09:41:05.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.922+0000 7efceb9a0700 1 -- 172.21.15.194:0/675143639 --> v1:172.21.15.194:6800/72864 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7efce411d830 con 0x7efcc80482b0 2024-04-26T09:41:05.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.922+0000 7efcd9ffb700 1 -- 172.21.15.194:0/675143639 <== osd.1 v1:172.21.15.194:6800/72864 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (unknown 0 0 1509541053) 0x7efce411d830 con 0x7efcc80482b0 2024-04-26T09:41:05.924 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-26T09:41:05.924 INFO:teuthology.orchestra.run.smithi194.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-04-26T09:41:05.924 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-26T09:41:05.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.922+0000 7efceb9a0700 1 -- 172.21.15.194:0/675143639 >> v1:172.21.15.194:6800/72864 conn(0x7efcc80482b0 legacy=0x7efcc804a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.922+0000 7efceb9a0700 1 -- 172.21.15.194:0/675143639 >> v1:172.21.15.194:6816/72244 conn(0x7efcc8041b80 legacy=0x7efcc8044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.922+0000 7efceb9a0700 1 -- 172.21.15.194:0/675143639 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efce408e750 msgr2=0x7efce408baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:05.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.922+0000 7efceb9a0700 1 --2- 172.21.15.194:0/675143639 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7efce408e750 0x7efce408baa0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7efcd400ec80 tx=0x7efcd4006f40).stop 2024-04-26T09:41:05.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.922+0000 7efce9f3d700 1 -- 172.21.15.194:0/675143639 reap_dead start 2024-04-26T09:41:05.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.923+0000 7efceb9a0700 1 -- 172.21.15.194:0/675143639 shutdown_connections 2024-04-26T09:41:05.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.923+0000 7efceb9a0700 1 -- 172.21.15.194:0/675143639 >> 172.21.15.194:0/675143639 conn(0x7efce4094a70 msgr2=0x7efce40940c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:05.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.923+0000 7efceb9a0700 1 -- 172.21.15.194:0/675143639 shutdown_connections 2024-04-26T09:41:05.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:05.923+0000 7efceb9a0700 1 -- 172.21.15.194:0/675143639 wait complete. 2024-04-26T09:41:05.935 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-26T09:41:06.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.048+0000 7fb0a66b8700 1 Processor -- start 2024-04-26T09:41:06.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.048+0000 7fb0a66b8700 1 -- start start 2024-04-26T09:41:06.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.048+0000 7fb0a66b8700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a008c4d0 0x7fb0a008c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb0a66b8700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008cda0 0x7fb0a0119d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb0a66b8700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a008d950 con 0x7fb0a011a240 2024-04-26T09:41:06.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb0a66b8700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a008d200 con 0x7fb0a008cda0 2024-04-26T09:41:06.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb0a66b8700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a008dbb0 con 0x7fb0a008c4d0 2024-04-26T09:41:06.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09f7fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008cda0 0x7fb0a0119d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09ffff700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a008c4d0 0x7fb0a008c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09ffff700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a008c4d0 0x7fb0a008c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47792/0 (socket says 172.21.15.194:47792) 2024-04-26T09:41:06.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09f7fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008cda0 0x7fb0a0119d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56472/0 (socket says 172.21.15.194:56472) 2024-04-26T09:41:06.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09ffff700 1 -- 172.21.15.194:0/2691567054 learned_addr learned my addr 172.21.15.194:0/2691567054 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:06.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb0a4c55700 1 --1- 172.21.15.194:0/2691567054 >> v1:172.21.15.194:6789/0 conn(0x7fb0a011a240 0x7fb0a012a660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:52916/0 (socket says 172.21.15.194:52916) 2024-04-26T09:41:06.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09effd700 1 -- 172.21.15.194:0/2691567054 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303174242 0 0) 0x7fb0a008d950 con 0x7fb0a011a240 2024-04-26T09:41:06.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09f7fe700 1 -- 172.21.15.194:0/2691567054 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a008c4d0 msgr2=0x7fb0a008c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:06.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09f7fe700 1 --2- 172.21.15.194:0/2691567054 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a008c4d0 0x7fb0a008c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09f7fe700 1 -- 172.21.15.194:0/2691567054 >> v1:172.21.15.194:6789/0 conn(0x7fb0a011a240 legacy=0x7fb0a012a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09f7fe700 1 -- 172.21.15.194:0/2691567054 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a008c2d0 con 0x7fb0a008cda0 2024-04-26T09:41:06.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09ffff700 1 --2- 172.21.15.194:0/2691567054 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a008c4d0 0x7fb0a008c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09f7fe700 1 --2- 172.21.15.194:0/2691567054 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008cda0 0x7fb0a0119d50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb090009790 tx=0x7fb090009060).ready entity=mon.1 client_cookie=8b08689ffdfa7868 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:06.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09effd700 1 -- 172.21.15.194:0/2691567054 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0900090d0 con 0x7fb0a008cda0 2024-04-26T09:41:06.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09effd700 1 -- 172.21.15.194:0/2691567054 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09000f510 con 0x7fb0a008cda0 2024-04-26T09:41:06.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.049+0000 7fb09effd700 1 -- 172.21.15.194:0/2691567054 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09000f8e0 con 0x7fb0a008cda0 2024-04-26T09:41:06.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.050+0000 7fb0a66b8700 1 -- 172.21.15.194:0/2691567054 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008cda0 msgr2=0x7fb0a0119d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.050+0000 7fb0a66b8700 1 --2- 172.21.15.194:0/2691567054 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008cda0 0x7fb0a0119d50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fb090009790 tx=0x7fb090009060).stop 2024-04-26T09:41:06.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.050+0000 7fb0a66b8700 1 -- 172.21.15.194:0/2691567054 shutdown_connections 2024-04-26T09:41:06.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.050+0000 7fb0a66b8700 1 --2- 172.21.15.194:0/2691567054 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a008c4d0 0x7fb0a008c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.050+0000 7fb0a66b8700 1 --2- 172.21.15.194:0/2691567054 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008cda0 0x7fb0a0119d50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.050+0000 7fb0a66b8700 1 -- 172.21.15.194:0/2691567054 >> 172.21.15.194:0/2691567054 conn(0x7fb0a0091b80 msgr2=0x7fb0a0091f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:06.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.050+0000 7fb0a66b8700 1 -- 172.21.15.194:0/2691567054 shutdown_connections 2024-04-26T09:41:06.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.050+0000 7fb0a66b8700 1 -- 172.21.15.194:0/2691567054 wait complete. 2024-04-26T09:41:06.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb0a66b8700 1 Processor -- start 2024-04-26T09:41:06.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb0a66b8700 1 -- start start 2024-04-26T09:41:06.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb0a66b8700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008c4d0 0x7fb0a0132ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb0a66b8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0a008cda0 0x7fb0a0133090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb0a66b8700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a011a240 0x7fb0a0133580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb0a66b8700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0133a70 con 0x7fb0a008cda0 2024-04-26T09:41:06.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb0a66b8700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0122a50 con 0x7fb0a008c4d0 2024-04-26T09:41:06.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb0a66b8700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a0124bd0 con 0x7fb0a011a240 2024-04-26T09:41:06.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb0a4c55700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a011a240 0x7fb0a0133580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb09f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0a008cda0 0x7fb0a0133090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb09ffff700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008c4d0 0x7fb0a0132ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb09f7fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0a008cda0 0x7fb0a0133090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44516/0 (socket says 172.21.15.194:44516) 2024-04-26T09:41:06.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb0a4c55700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a011a240 0x7fb0a0133580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47794/0 (socket says 172.21.15.194:47794) 2024-04-26T09:41:06.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.051+0000 7fb09f7fe700 1 -- 172.21.15.194:0/1627726350 learned_addr learned my addr 172.21.15.194:0/1627726350 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:06.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb09ffff700 1 -- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a011a240 msgr2=0x7fb0a0133580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb09ffff700 1 --2- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a011a240 0x7fb0a0133580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb09ffff700 1 -- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0a008cda0 msgr2=0x7fb0a0133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb09ffff700 1 --2- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0a008cda0 0x7fb0a0133090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb09ffff700 1 -- 172.21.15.194:0/1627726350 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb094009300 con 0x7fb0a008c4d0 2024-04-26T09:41:06.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb0a4c55700 1 --2- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a011a240 0x7fb0a0133580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb09f7fe700 1 --2- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0a008cda0 0x7fb0a0133090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb09ffff700 1 --2- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008c4d0 0x7fb0a0132ba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb09400e950 tx=0x7fb094009060).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:06.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb09cff9700 1 -- 172.21.15.194:0/1627726350 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094009580 con 0x7fb0a008c4d0 2024-04-26T09:41:06.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb09cff9700 1 -- 172.21.15.194:0/1627726350 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb094005880 con 0x7fb0a008c4d0 2024-04-26T09:41:06.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb0a66b8700 1 -- 172.21.15.194:0/1627726350 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0900092a0 con 0x7fb0a008c4d0 2024-04-26T09:41:06.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb09cff9700 1 -- 172.21.15.194:0/1627726350 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094005c00 con 0x7fb0a008c4d0 2024-04-26T09:41:06.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.052+0000 7fb0a66b8700 1 -- 172.21.15.194:0/1627726350 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a0121810 con 0x7fb0a008c4d0 2024-04-26T09:41:06.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.053+0000 7fb09cff9700 1 -- 172.21.15.194:0/1627726350 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fb094005f70 con 0x7fb0a008c4d0 2024-04-26T09:41:06.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.053+0000 7fb0a66b8700 1 -- 172.21.15.194:0/1627726350 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a01233a0 con 0x7fb0a008c4d0 2024-04-26T09:41:06.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.053+0000 7fb09cff9700 1 -- 172.21.15.194:0/1627726350 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fb094055130 con 0x7fb0a008c4d0 2024-04-26T09:41:06.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.057+0000 7fb09cff9700 1 -- 172.21.15.194:0/1627726350 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fb094035b30 con 0x7fb0a008c4d0 2024-04-26T09:41:06.200 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.198+0000 7fb0a66b8700 1 -- 172.21.15.194:0/1627726350 --> v1:172.21.15.194:6816/72244 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0a01246a0 con 0x7fb08c041b80 2024-04-26T09:41:06.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.198+0000 7fb09cff9700 1 -- 172.21.15.194:0/1627726350 <== mgr.4111 v1:172.21.15.194:6816/72244 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (unknown 489605429 0 0) 0x7fb0900090d0 con 0x7fb08c041b80 2024-04-26T09:41:06.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.200+0000 7fb0a66b8700 1 -- 172.21.15.194:0/1627726350 >> v1:172.21.15.194:6816/72244 conn(0x7fb08c041b80 legacy=0x7fb08c044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.200+0000 7fb0a66b8700 1 -- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008c4d0 msgr2=0x7fb0a0132ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.200+0000 7fb0a66b8700 1 --2- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008c4d0 0x7fb0a0132ba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb09400e950 tx=0x7fb094009060).stop 2024-04-26T09:41:06.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.200+0000 7fb0a66b8700 1 -- 172.21.15.194:0/1627726350 shutdown_connections 2024-04-26T09:41:06.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.200+0000 7fb0a66b8700 1 --2- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb0a011a240 0x7fb0a0133580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.200+0000 7fb0a66b8700 1 --2- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb0a008c4d0 0x7fb0a0132ba0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.200+0000 7fb0a66b8700 1 --2- 172.21.15.194:0/1627726350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0a008cda0 0x7fb0a0133090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.200+0000 7fb0a66b8700 1 -- 172.21.15.194:0/1627726350 >> 172.21.15.194:0/1627726350 conn(0x7fb0a0091b80 msgr2=0x7fb0a0067880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:06.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.200+0000 7fb0a66b8700 1 -- 172.21.15.194:0/1627726350 shutdown_connections 2024-04-26T09:41:06.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.200+0000 7fb0a66b8700 1 -- 172.21.15.194:0/1627726350 wait complete. 2024-04-26T09:41:06.204 INFO:teuthology.orchestra.run.smithi194.stderr:instructed osd(s) 1 to deep-scrub 2024-04-26T09:41:06.214 INFO:tasks.ceph:Scrubbing osd.2 2024-04-26T09:41:06.214 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-26T09:41:06.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.330+0000 7fb2d9db4700 1 Processor -- start 2024-04-26T09:41:06.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.330+0000 7fb2d9db4700 1 -- start start 2024-04-26T09:41:06.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.330+0000 7fb2d9db4700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb2d408ecb0 0x7fb2d40904a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d9db4700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d4090990 0x7fb2d4090db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d9db4700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d4066290 con 0x7fb2d4129c80 2024-04-26T09:41:06.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d9db4700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d4066a20 con 0x7fb2d408ecb0 2024-04-26T09:41:06.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d9db4700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d40912a0 con 0x7fb2d4090990 2024-04-26T09:41:06.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d2ffd700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d4090990 0x7fb2d4090db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d37fe700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb2d408ecb0 0x7fb2d40904a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d2ffd700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d4090990 0x7fb2d4090db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56488/0 (socket says 172.21.15.194:56488) 2024-04-26T09:41:06.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d3fff700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fb2d4129c80 0x7fb2d412c070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:52928/0 (socket says 172.21.15.194:52928) 2024-04-26T09:41:06.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d2ffd700 1 -- 172.21.15.194:0/2142806056 learned_addr learned my addr 172.21.15.194:0/2142806056 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:06.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d37fe700 1 --2- 172.21.15.194:0/2142806056 >> v2:172.21.15.194:3300/0 conn(0x7fb2d408ecb0 0x7fb2d40904a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44528/0 (socket says 172.21.15.194:44528) 2024-04-26T09:41:06.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d37fe700 1 -- 172.21.15.194:0/2142806056 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d4090990 msgr2=0x7fb2d4090db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:06.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d37fe700 1 --2- 172.21.15.194:0/2142806056 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d4090990 0x7fb2d4090db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d37fe700 1 -- 172.21.15.194:0/2142806056 >> v1:172.21.15.194:6789/0 conn(0x7fb2d4129c80 legacy=0x7fb2d412c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d37fe700 1 -- 172.21.15.194:0/2142806056 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d405ee70 con 0x7fb2d408ecb0 2024-04-26T09:41:06.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d27fc700 1 -- 172.21.15.194:0/2142806056 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557940777 0 0) 0x7fb2d4066290 con 0x7fb2d4129c80 2024-04-26T09:41:06.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d2ffd700 1 --2- 172.21.15.194:0/2142806056 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d4090990 0x7fb2d4090db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d37fe700 1 --2- 172.21.15.194:0/2142806056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2d408ecb0 0x7fb2d40904a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fb2cc006fd0 tx=0x7fb2cc006f40).ready entity=mon.0 client_cookie=b77de8f0789b306d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:06.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d27fc700 1 -- 172.21.15.194:0/2142806056 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc005880 con 0x7fb2d408ecb0 2024-04-26T09:41:06.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d27fc700 1 -- 172.21.15.194:0/2142806056 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2cc0059e0 con 0x7fb2d408ecb0 2024-04-26T09:41:06.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.331+0000 7fb2d27fc700 1 -- 172.21.15.194:0/2142806056 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc005d60 con 0x7fb2d408ecb0 2024-04-26T09:41:06.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.332+0000 7fb2d9db4700 1 -- 172.21.15.194:0/2142806056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2d408ecb0 msgr2=0x7fb2d40904a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.332+0000 7fb2d9db4700 1 --2- 172.21.15.194:0/2142806056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2d408ecb0 0x7fb2d40904a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fb2cc006fd0 tx=0x7fb2cc006f40).stop 2024-04-26T09:41:06.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.332+0000 7fb2d9db4700 1 -- 172.21.15.194:0/2142806056 shutdown_connections 2024-04-26T09:41:06.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.332+0000 7fb2d9db4700 1 --2- 172.21.15.194:0/2142806056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2d408ecb0 0x7fb2d40904a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.332+0000 7fb2d9db4700 1 --2- 172.21.15.194:0/2142806056 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d4090990 0x7fb2d4090db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.332+0000 7fb2d9db4700 1 -- 172.21.15.194:0/2142806056 >> 172.21.15.194:0/2142806056 conn(0x7fb2d4094a70 msgr2=0x7fb2d4094e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:06.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.332+0000 7fb2d9db4700 1 -- 172.21.15.194:0/2142806056 shutdown_connections 2024-04-26T09:41:06.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.332+0000 7fb2d9db4700 1 -- 172.21.15.194:0/2142806056 wait complete. 2024-04-26T09:41:06.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d9db4700 1 Processor -- start 2024-04-26T09:41:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d9db4700 1 -- start start 2024-04-26T09:41:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d9db4700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d408ecb0 0x7fb2d411a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d9db4700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2d4090990 0x7fb2d411abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d9db4700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb2d4129c80 0x7fb2d411b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d9db4700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d4069c00 con 0x7fb2d4090990 2024-04-26T09:41:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d9db4700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d411da60 con 0x7fb2d408ecb0 2024-04-26T09:41:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d9db4700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d411dd70 con 0x7fb2d4129c80 2024-04-26T09:41:06.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d37fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d408ecb0 0x7fb2d411a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d2ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2d4090990 0x7fb2d411abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d37fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d408ecb0 0x7fb2d411a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56504/0 (socket says 172.21.15.194:56504) 2024-04-26T09:41:06.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d37fe700 1 -- 172.21.15.194:0/3890631520 learned_addr learned my addr 172.21.15.194:0/3890631520 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:06.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.333+0000 7fb2d3fff700 1 --2- 172.21.15.194:0/3890631520 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb2d4129c80 0x7fb2d411b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2d2ffd700 1 -- 172.21.15.194:0/3890631520 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb2d4129c80 msgr2=0x7fb2d411b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2d2ffd700 1 --2- 172.21.15.194:0/3890631520 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fb2d4129c80 0x7fb2d411b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2d2ffd700 1 -- 172.21.15.194:0/3890631520 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d408ecb0 msgr2=0x7fb2d411a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2d2ffd700 1 --2- 172.21.15.194:0/3890631520 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d408ecb0 0x7fb2d411a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2d2ffd700 1 -- 172.21.15.194:0/3890631520 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2bc009370 con 0x7fb2d4090990 2024-04-26T09:41:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2d37fe700 1 --2- 172.21.15.194:0/3890631520 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fb2d408ecb0 0x7fb2d411a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2d2ffd700 1 --2- 172.21.15.194:0/3890631520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2d4090990 0x7fb2d411abc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fb2bc006fd0 tx=0x7fb2bc0090f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2cbfff700 1 -- 172.21.15.194:0/3890631520 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc009650 con 0x7fb2d4090990 2024-04-26T09:41:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2cbfff700 1 -- 172.21.15.194:0/3890631520 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc00c6a0 con 0x7fb2d4090990 2024-04-26T09:41:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2d9db4700 1 -- 172.21.15.194:0/3890631520 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc009250 con 0x7fb2d4090990 2024-04-26T09:41:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2cbfff700 1 -- 172.21.15.194:0/3890631520 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc00ca20 con 0x7fb2d4090990 2024-04-26T09:41:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.334+0000 7fb2d9db4700 1 -- 172.21.15.194:0/3890631520 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d411b5a0 con 0x7fb2d4090990 2024-04-26T09:41:06.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.335+0000 7fb2d9db4700 1 -- 172.21.15.194:0/3890631520 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2d411b7f0 con 0x7fb2d4090990 2024-04-26T09:41:06.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.335+0000 7fb2cbfff700 1 -- 172.21.15.194:0/3890631520 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fb2bc01b400 con 0x7fb2d4090990 2024-04-26T09:41:06.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.335+0000 7fb2cbfff700 1 -- 172.21.15.194:0/3890631520 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fb2bc055660 con 0x7fb2d4090990 2024-04-26T09:41:06.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.335+0000 7fb2cbfff700 1 -- 172.21.15.194:0/3890631520 --> v1:172.21.15.194:6812/72901 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2b404bfa0 con 0x7fb2b40482b0 2024-04-26T09:41:06.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.335+0000 7fb2cbfff700 1 -- 172.21.15.194:0/3890631520 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2bc057c80 con 0x7fb2d4090990 2024-04-26T09:41:06.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.336+0000 7fb2cbfff700 1 -- 172.21.15.194:0/3890631520 <== osd.2 v1:172.21.15.194:6812/72901 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7fb2b404bfa0 con 0x7fb2b40482b0 2024-04-26T09:41:06.351 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.349+0000 7fb2d9db4700 1 -- 172.21.15.194:0/3890631520 --> v1:172.21.15.194:6812/72901 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb2d411d460 con 0x7fb2b40482b0 2024-04-26T09:41:06.352 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.349+0000 7fb2cbfff700 1 -- 172.21.15.194:0/3890631520 <== osd.2 v1:172.21.15.194:6812/72901 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (unknown 0 0 1509541053) 0x7fb2d411d460 con 0x7fb2b40482b0 2024-04-26T09:41:06.352 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-26T09:41:06.352 INFO:teuthology.orchestra.run.smithi194.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-04-26T09:41:06.352 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-26T09:41:06.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.349+0000 7fb2d9db4700 1 -- 172.21.15.194:0/3890631520 >> v1:172.21.15.194:6812/72901 conn(0x7fb2b40482b0 legacy=0x7fb2b404a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.350+0000 7fb2d9db4700 1 -- 172.21.15.194:0/3890631520 >> v1:172.21.15.194:6816/72244 conn(0x7fb2b4041b80 legacy=0x7fb2b4044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.350+0000 7fb2d9db4700 1 -- 172.21.15.194:0/3890631520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2d4090990 msgr2=0x7fb2d411abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.350+0000 7fb2d9db4700 1 --2- 172.21.15.194:0/3890631520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2d4090990 0x7fb2d411abc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fb2bc006fd0 tx=0x7fb2bc0090f0).stop 2024-04-26T09:41:06.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.350+0000 7fb2d3fff700 1 -- 172.21.15.194:0/3890631520 reap_dead start 2024-04-26T09:41:06.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.350+0000 7fb2d9db4700 1 -- 172.21.15.194:0/3890631520 shutdown_connections 2024-04-26T09:41:06.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.350+0000 7fb2d9db4700 1 -- 172.21.15.194:0/3890631520 >> 172.21.15.194:0/3890631520 conn(0x7fb2d4094a70 msgr2=0x7fb2d40687d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:06.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.350+0000 7fb2d9db4700 1 -- 172.21.15.194:0/3890631520 shutdown_connections 2024-04-26T09:41:06.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.350+0000 7fb2d9db4700 1 -- 172.21.15.194:0/3890631520 wait complete. 2024-04-26T09:41:06.363 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-26T09:41:06.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.475+0000 7f3c11893700 1 Processor -- start 2024-04-26T09:41:06.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.475+0000 7f3c11893700 1 -- start start 2024-04-26T09:41:06.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.475+0000 7f3c11893700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c063900 0x7f3c0c11fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.475+0000 7f3c11893700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f3c0c069ab0 0x7f3c0c120260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c11893700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c123020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c11893700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0c11fb40 con 0x7f3c0c069ab0 2024-04-26T09:41:06.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c11893700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c10bc20 con 0x7f3c0c120750 2024-04-26T09:41:06.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c11893700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c10bd90 con 0x7f3c0c063900 2024-04-26T09:41:06.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0b7fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c123020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0a7fc700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f3c0c069ab0 0x7f3c0c120260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0affd700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c063900 0x7f3c0c11fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0b7fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c123020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56520/0 (socket says 172.21.15.194:56520) 2024-04-26T09:41:06.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0a7fc700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f3c0c069ab0 0x7f3c0c120260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44544/0 (socket says 172.21.15.194:44544) 2024-04-26T09:41:06.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0b7fe700 1 -- 172.21.15.194:0/447607930 learned_addr learned my addr 172.21.15.194:0/447607930 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:06.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0b7fe700 1 -- 172.21.15.194:0/447607930 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c063900 msgr2=0x7f3c0c11fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:06.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0b7fe700 1 --2- 172.21.15.194:0/447607930 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c063900 0x7f3c0c11fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0b7fe700 1 -- 172.21.15.194:0/447607930 >> v2:172.21.15.194:3300/0 conn(0x7f3c0c069ab0 msgr2=0x7f3c0c120260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:06.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0b7fe700 1 --2- 172.21.15.194:0/447607930 >> v2:172.21.15.194:3300/0 conn(0x7f3c0c069ab0 0x7f3c0c120260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0b7fe700 1 -- 172.21.15.194:0/447607930 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c123510 con 0x7f3c0c120750 2024-04-26T09:41:06.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0affd700 1 --2- 172.21.15.194:0/447607930 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c063900 0x7f3c0c11fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0a7fc700 1 --2- 172.21.15.194:0/447607930 >> v2:172.21.15.194:3300/0 conn(0x7f3c0c069ab0 0x7f3c0c120260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c0b7fe700 1 --2- 172.21.15.194:0/447607930 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c123020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f3c04009f50 tx=0x7f3c04009840).ready entity=mon.1 client_cookie=e3bf8105d464d7d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:06.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c09ffb700 1 -- 172.21.15.194:0/447607930 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0400e070 con 0x7f3c0c120750 2024-04-26T09:41:06.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c09ffb700 1 -- 172.21.15.194:0/447607930 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0400b910 con 0x7f3c0c120750 2024-04-26T09:41:06.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.476+0000 7f3c09ffb700 1 -- 172.21.15.194:0/447607930 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0400bc90 con 0x7f3c0c120750 2024-04-26T09:41:06.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.477+0000 7f3c11893700 1 -- 172.21.15.194:0/447607930 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 msgr2=0x7f3c0c123020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.477+0000 7f3c11893700 1 --2- 172.21.15.194:0/447607930 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c123020 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f3c04009f50 tx=0x7f3c04009840).stop 2024-04-26T09:41:06.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.477+0000 7f3c11893700 1 -- 172.21.15.194:0/447607930 shutdown_connections 2024-04-26T09:41:06.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.477+0000 7f3c11893700 1 --2- 172.21.15.194:0/447607930 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c123020 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.477+0000 7f3c11893700 1 --2- 172.21.15.194:0/447607930 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c063900 0x7f3c0c11fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.477+0000 7f3c11893700 1 --2- 172.21.15.194:0/447607930 >> v2:172.21.15.194:3300/0 conn(0x7f3c0c069ab0 0x7f3c0c120260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.477+0000 7f3c11893700 1 -- 172.21.15.194:0/447607930 >> 172.21.15.194:0/447607930 conn(0x7f3c0c095750 msgr2=0x7f3c0c095b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:06.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.477+0000 7f3c11893700 1 -- 172.21.15.194:0/447607930 shutdown_connections 2024-04-26T09:41:06.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.477+0000 7f3c11893700 1 -- 172.21.15.194:0/447607930 wait complete. 2024-04-26T09:41:06.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c11893700 1 Processor -- start 2024-04-26T09:41:06.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c11893700 1 -- start start 2024-04-26T09:41:06.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c11893700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c0c063900 0x7f3c0c1329f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c11893700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c069ab0 0x7f3c0c132ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c11893700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c1333d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c11893700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c1338c0 con 0x7f3c0c063900 2024-04-26T09:41:06.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c11893700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c11d580 con 0x7f3c0c120750 2024-04-26T09:41:06.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c11893700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c10e8e0 con 0x7f3c0c069ab0 2024-04-26T09:41:06.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c0affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c0c063900 0x7f3c0c1329f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c0b7fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c1333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c0affd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c0c063900 0x7f3c0c1329f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44560/0 (socket says 172.21.15.194:44560) 2024-04-26T09:41:06.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c0affd700 1 -- 172.21.15.194:0/1783747263 learned_addr learned my addr 172.21.15.194:0/1783747263 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:06.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c0b7fe700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c1333d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56530/0 (socket says 172.21.15.194:56530) 2024-04-26T09:41:06.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.478+0000 7f3c0a7fc700 1 --2- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c069ab0 0x7f3c0c132ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3c0affd700 1 -- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c069ab0 msgr2=0x7f3c0c132ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3c0affd700 1 --2- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c069ab0 0x7f3c0c132ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3c0affd700 1 -- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 msgr2=0x7f3c0c1333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3c0affd700 1 --2- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c1333d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3c0affd700 1 -- 172.21.15.194:0/1783747263 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c000092a0 con 0x7f3c0c063900 2024-04-26T09:41:06.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3c0a7fc700 1 --2- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c069ab0 0x7f3c0c132ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3c0b7fe700 1 --2- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c1333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3c0affd700 1 --2- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c0c063900 0x7f3c0c1329f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f3c0000ea20 tx=0x7f3c00009060).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:06.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3bf37fe700 1 -- 172.21.15.194:0/1783747263 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0000ee60 con 0x7f3c0c063900 2024-04-26T09:41:06.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3bf37fe700 1 -- 172.21.15.194:0/1783747263 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0000c850 con 0x7f3c0c063900 2024-04-26T09:41:06.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3c11893700 1 -- 172.21.15.194:0/1783747263 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc0092a0 con 0x7f3c0c063900 2024-04-26T09:41:06.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3bf37fe700 1 -- 172.21.15.194:0/1783747263 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0000cbd0 con 0x7f3c0c063900 2024-04-26T09:41:06.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.479+0000 7f3c11893700 1 -- 172.21.15.194:0/1783747263 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c04009ae0 con 0x7f3c0c063900 2024-04-26T09:41:06.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.480+0000 7f3c11893700 1 -- 172.21.15.194:0/1783747263 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0c11d3f0 con 0x7f3c0c063900 2024-04-26T09:41:06.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.480+0000 7f3bf37fe700 1 -- 172.21.15.194:0/1783747263 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f3c000123f0 con 0x7f3c0c063900 2024-04-26T09:41:06.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.480+0000 7f3bf37fe700 1 -- 172.21.15.194:0/1783747263 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f3c00055080 con 0x7f3c0c063900 2024-04-26T09:41:06.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.484+0000 7f3bf37fe700 1 -- 172.21.15.194:0/1783747263 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7f3c00035a20 con 0x7f3c0c063900 2024-04-26T09:41:06.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.625+0000 7f3c11893700 1 -- 172.21.15.194:0/1783747263 --> v1:172.21.15.194:6816/72244 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c0c123940 con 0x7f3bf8041b80 2024-04-26T09:41:06.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.625+0000 7f3bf37fe700 1 -- 172.21.15.194:0/1783747263 <== mgr.4111 v1:172.21.15.194:6816/72244 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (unknown 3049115190 0 0) 0x7f3c0c123940 con 0x7f3bf8041b80 2024-04-26T09:41:06.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.626+0000 7f3c11893700 1 -- 172.21.15.194:0/1783747263 >> v1:172.21.15.194:6816/72244 conn(0x7f3bf8041b80 legacy=0x7f3bf8044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.627+0000 7f3c11893700 1 -- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c0c063900 msgr2=0x7f3c0c1329f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.627+0000 7f3c11893700 1 --2- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c0c063900 0x7f3c0c1329f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f3c0000ea20 tx=0x7f3c00009060).stop 2024-04-26T09:41:06.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.627+0000 7f3c11893700 1 -- 172.21.15.194:0/1783747263 shutdown_connections 2024-04-26T09:41:06.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.627+0000 7f3c11893700 1 --2- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f3c0c120750 0x7f3c0c1333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.627+0000 7f3c11893700 1 --2- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c0c063900 0x7f3c0c1329f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.627+0000 7f3c11893700 1 --2- 172.21.15.194:0/1783747263 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f3c0c069ab0 0x7f3c0c132ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.627+0000 7f3c11893700 1 -- 172.21.15.194:0/1783747263 >> 172.21.15.194:0/1783747263 conn(0x7f3c0c095750 msgr2=0x7f3c0c120f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:06.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.627+0000 7f3c11893700 1 -- 172.21.15.194:0/1783747263 shutdown_connections 2024-04-26T09:41:06.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.627+0000 7f3c11893700 1 -- 172.21.15.194:0/1783747263 wait complete. 2024-04-26T09:41:06.631 INFO:teuthology.orchestra.run.smithi194.stderr:instructed osd(s) 2 to deep-scrub 2024-04-26T09:41:06.641 INFO:tasks.ceph:Scrubbing osd.3 2024-04-26T09:41:06.641 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-26T09:41:06.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.756+0000 7f0ddba1d700 1 Processor -- start 2024-04-26T09:41:06.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.756+0000 7f0ddba1d700 1 -- start start 2024-04-26T09:41:06.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0ddba1d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 0x7f0dd4090de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0ddba1d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0dd4123060 0x7f0dd41234c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0ddba1d700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd4065cf0 con 0x7f0dd41239b0 2024-04-26T09:41:06.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0ddba1d700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40959c0 con 0x7f0dd4123060 2024-04-26T09:41:06.761 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0ddba1d700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd406a150 con 0x7f0dd4090a00 2024-04-26T09:41:06.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dd97b9700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 0x7f0dd4090de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dd8fb8700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0dd4123060 0x7f0dd41234c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dd9fba700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f0dd41239b0 0x7f0dd412a3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:52938/0 (socket says 172.21.15.194:52938) 2024-04-26T09:41:06.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dd9fba700 1 -- 172.21.15.194:0/263424549 learned_addr learned my addr 172.21.15.194:0/263424549 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:41:06.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dcbfff700 1 -- 172.21.15.194:0/263424549 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941252185 0 0) 0x7f0dd4065cf0 con 0x7f0dd41239b0 2024-04-26T09:41:06.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dd97b9700 1 -- 172.21.15.194:0/263424549 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0dd4123060 msgr2=0x7f0dd41234c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:06.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dd97b9700 1 --2- 172.21.15.194:0/263424549 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0dd4123060 0x7f0dd41234c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dd97b9700 1 -- 172.21.15.194:0/263424549 >> v1:172.21.15.194:6789/0 conn(0x7f0dd41239b0 legacy=0x7f0dd412a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dd97b9700 1 -- 172.21.15.194:0/263424549 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd4068de0 con 0x7f0dd4090a00 2024-04-26T09:41:06.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dd8fb8700 1 --2- 172.21.15.194:0/263424549 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0dd4123060 0x7f0dd41234c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dd97b9700 1 --2- 172.21.15.194:0/263424549 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 0x7f0dd4090de0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0dd0009790 tx=0x7f0dd0006f40).ready entity=mon.2 client_cookie=3bc47c44d263fb8d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:06.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dcbfff700 1 -- 172.21.15.194:0/263424549 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0005880 con 0x7f0dd4090a00 2024-04-26T09:41:06.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dcbfff700 1 -- 172.21.15.194:0/263424549 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd00059e0 con 0x7f0dd4090a00 2024-04-26T09:41:06.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.757+0000 7f0dcbfff700 1 -- 172.21.15.194:0/263424549 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0005d60 con 0x7f0dd4090a00 2024-04-26T09:41:06.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.758+0000 7f0ddba1d700 1 -- 172.21.15.194:0/263424549 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 msgr2=0x7f0dd4090de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.758+0000 7f0ddba1d700 1 --2- 172.21.15.194:0/263424549 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 0x7f0dd4090de0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0dd0009790 tx=0x7f0dd0006f40).stop 2024-04-26T09:41:06.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.758+0000 7f0ddba1d700 1 -- 172.21.15.194:0/263424549 shutdown_connections 2024-04-26T09:41:06.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.758+0000 7f0ddba1d700 1 --2- 172.21.15.194:0/263424549 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 0x7f0dd4090de0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.758+0000 7f0ddba1d700 1 --2- 172.21.15.194:0/263424549 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0dd4123060 0x7f0dd41234c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.758+0000 7f0ddba1d700 1 -- 172.21.15.194:0/263424549 >> 172.21.15.194:0/263424549 conn(0x7f0dd4094a70 msgr2=0x7f0dd4094e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:06.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.758+0000 7f0ddba1d700 1 -- 172.21.15.194:0/263424549 shutdown_connections 2024-04-26T09:41:06.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.758+0000 7f0ddba1d700 1 -- 172.21.15.194:0/263424549 wait complete. 2024-04-26T09:41:06.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0ddba1d700 1 Processor -- start 2024-04-26T09:41:06.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0ddba1d700 1 -- start start 2024-04-26T09:41:06.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0ddba1d700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 0x7f0dd41329a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0ddba1d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dd4123060 0x7f0dd4132e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0ddba1d700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0dd41239b0 0x7f0dd411daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0ddba1d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd40838e0 con 0x7f0dd4123060 2024-04-26T09:41:06.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0ddba1d700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd4083250 con 0x7f0dd41239b0 2024-04-26T09:41:06.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0ddba1d700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd4083620 con 0x7f0dd4090a00 2024-04-26T09:41:06.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0dd97b9700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 0x7f0dd41329a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0dd97b9700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 0x7f0dd41329a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47844/0 (socket says 172.21.15.194:47844) 2024-04-26T09:41:06.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0dd8fb8700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dd4123060 0x7f0dd4132e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0dd97b9700 1 -- 172.21.15.194:0/3002949075 learned_addr learned my addr 172.21.15.194:0/3002949075 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:06.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.759+0000 7f0dd9fba700 1 --2- 172.21.15.194:0/3002949075 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0dd41239b0 0x7f0dd411daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0dd9fba700 1 -- 172.21.15.194:0/3002949075 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 msgr2=0x7f0dd41329a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0dd9fba700 1 --2- 172.21.15.194:0/3002949075 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 0x7f0dd41329a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0dd9fba700 1 -- 172.21.15.194:0/3002949075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dd4123060 msgr2=0x7f0dd4132e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0dd9fba700 1 --2- 172.21.15.194:0/3002949075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dd4123060 0x7f0dd4132e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0dd9fba700 1 -- 172.21.15.194:0/3002949075 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc40092a0 con 0x7f0dd41239b0 2024-04-26T09:41:06.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0dd97b9700 1 --2- 172.21.15.194:0/3002949075 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7f0dd4090a00 0x7f0dd41329a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0dd8fb8700 1 --2- 172.21.15.194:0/3002949075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0dd4123060 0x7f0dd4132e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0dd9fba700 1 --2- 172.21.15.194:0/3002949075 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0dd41239b0 0x7f0dd411daa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0dcc002230 tx=0x7f0dcc007eb0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:06.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0dc9ffb700 1 -- 172.21.15.194:0/3002949075 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc005d30 con 0x7f0dd41239b0 2024-04-26T09:41:06.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0dc9ffb700 1 -- 172.21.15.194:0/3002949075 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dcc007110 con 0x7f0dd41239b0 2024-04-26T09:41:06.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0ddba1d700 1 -- 172.21.15.194:0/3002949075 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd0009250 con 0x7f0dd41239b0 2024-04-26T09:41:06.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0dc9ffb700 1 -- 172.21.15.194:0/3002949075 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dcc0074d0 con 0x7f0dd41239b0 2024-04-26T09:41:06.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.760+0000 7f0ddba1d700 1 -- 172.21.15.194:0/3002949075 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd4133380 con 0x7f0dd41239b0 2024-04-26T09:41:06.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.761+0000 7f0ddba1d700 1 -- 172.21.15.194:0/3002949075 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0dd40923d0 con 0x7f0dd41239b0 2024-04-26T09:41:06.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.761+0000 7f0dc9ffb700 1 -- 172.21.15.194:0/3002949075 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7f0dcc013410 con 0x7f0dd41239b0 2024-04-26T09:41:06.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.761+0000 7f0dc9ffb700 1 -- 172.21.15.194:0/3002949075 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7f0dcc055100 con 0x7f0dd41239b0 2024-04-26T09:41:06.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.761+0000 7f0dc9ffb700 1 -- 172.21.15.194:0/3002949075 --> v1:172.21.15.194:6804/72866 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0db804bfa0 con 0x7f0db80482b0 2024-04-26T09:41:06.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.761+0000 7f0dc9ffb700 1 -- 172.21.15.194:0/3002949075 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_get_version_reply(handle=1 version=13) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0dcc057720 con 0x7f0dd41239b0 2024-04-26T09:41:06.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.762+0000 7f0dc9ffb700 1 -- 172.21.15.194:0/3002949075 <== osd.3 v1:172.21.15.194:6804/72866 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23213 (unknown 0 0 1671252174) 0x7f0db804bfa0 con 0x7f0db80482b0 2024-04-26T09:41:06.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.775+0000 7f0ddba1d700 1 -- 172.21.15.194:0/3002949075 --> v1:172.21.15.194:6804/72866 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0dd41208d0 con 0x7f0db80482b0 2024-04-26T09:41:06.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.775+0000 7f0dc9ffb700 1 -- 172.21.15.194:0/3002949075 <== osd.3 v1:172.21.15.194:6804/72866 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (unknown 0 0 1509541053) 0x7f0dd41208d0 con 0x7f0db80482b0 2024-04-26T09:41:06.778 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-26T09:41:06.778 INFO:teuthology.orchestra.run.smithi194.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2024-04-26T09:41:06.778 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-26T09:41:06.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.775+0000 7f0ddba1d700 1 -- 172.21.15.194:0/3002949075 >> v1:172.21.15.194:6804/72866 conn(0x7f0db80482b0 legacy=0x7f0db804a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.775+0000 7f0ddba1d700 1 -- 172.21.15.194:0/3002949075 >> v1:172.21.15.194:6816/72244 conn(0x7f0db8041b80 legacy=0x7f0db8044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.775+0000 7f0ddba1d700 1 -- 172.21.15.194:0/3002949075 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0dd41239b0 msgr2=0x7f0dd411daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.775+0000 7f0ddba1d700 1 --2- 172.21.15.194:0/3002949075 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7f0dd41239b0 0x7f0dd411daa0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f0dcc002230 tx=0x7f0dcc007eb0).stop 2024-04-26T09:41:06.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.775+0000 7f0dd9fba700 1 -- 172.21.15.194:0/3002949075 reap_dead start 2024-04-26T09:41:06.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.775+0000 7f0ddba1d700 1 -- 172.21.15.194:0/3002949075 shutdown_connections 2024-04-26T09:41:06.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.775+0000 7f0ddba1d700 1 -- 172.21.15.194:0/3002949075 >> 172.21.15.194:0/3002949075 conn(0x7f0dd4094a70 msgr2=0x7f0dd40928a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:06.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.775+0000 7f0ddba1d700 1 -- 172.21.15.194:0/3002949075 shutdown_connections 2024-04-26T09:41:06.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.776+0000 7f0ddba1d700 1 -- 172.21.15.194:0/3002949075 wait complete. 2024-04-26T09:41:06.789 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-26T09:41:06.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.901+0000 7ff98f81a700 1 Processor -- start 2024-04-26T09:41:06.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.901+0000 7ff98f81a700 1 -- start start 2024-04-26T09:41:06.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.901+0000 7ff98f81a700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7ff98808ecb0 0x7ff9880903d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.901+0000 7ff98f81a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff98805e230 0x7ff9880908c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.901+0000 7ff98f81a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff988090db0 0x7ff98812bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.901+0000 7ff98f81a700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff988066d70 con 0x7ff98808ecb0 2024-04-26T09:41:06.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.901+0000 7ff98f81a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98805f9c0 con 0x7ff988090db0 2024-04-26T09:41:06.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98f81a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff988091220 con 0x7ff98805e230 2024-04-26T09:41:06.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98cdb5700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff98805e230 0x7ff9880908c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98d5b6700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7ff98808ecb0 0x7ff9880903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98ddb7700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff988090db0 0x7ff98812bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98d5b6700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7ff98808ecb0 0x7ff9880903d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44574/0 (socket says 172.21.15.194:44574) 2024-04-26T09:41:06.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98cdb5700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff98805e230 0x7ff9880908c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47858/0 (socket says 172.21.15.194:47858) 2024-04-26T09:41:06.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98ddb7700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff988090db0 0x7ff98812bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:56558/0 (socket says 172.21.15.194:56558) 2024-04-26T09:41:06.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98d5b6700 1 -- 172.21.15.194:0/1199606348 learned_addr learned my addr 172.21.15.194:0/1199606348 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:06.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98d5b6700 1 -- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff98805e230 msgr2=0x7ff9880908c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:06.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98d5b6700 1 --2- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff98805e230 0x7ff9880908c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98d5b6700 1 -- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff988090db0 msgr2=0x7ff98812bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:06.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98d5b6700 1 --2- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff988090db0 0x7ff98812bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98d5b6700 1 -- 172.21.15.194:0/1199606348 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98805e8c0 con 0x7ff98808ecb0 2024-04-26T09:41:06.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98cdb5700 1 --2- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff98805e230 0x7ff9880908c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98ddb7700 1 --2- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff988090db0 0x7ff98812bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff98d5b6700 1 --2- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff98808ecb0 0x7ff9880903d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff978009810 tx=0x7ff978009060).ready entity=mon.0 client_cookie=e0cfa23c3c0d5214 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:06.910 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff97ffff700 1 -- 172.21.15.194:0/1199606348 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978005880 con 0x7ff98808ecb0 2024-04-26T09:41:06.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff97ffff700 1 -- 172.21.15.194:0/1199606348 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9780059e0 con 0x7ff98808ecb0 2024-04-26T09:41:06.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.902+0000 7ff97ffff700 1 -- 172.21.15.194:0/1199606348 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978005d60 con 0x7ff98808ecb0 2024-04-26T09:41:06.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.903+0000 7ff98f81a700 1 -- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff98808ecb0 msgr2=0x7ff9880903d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.903+0000 7ff98f81a700 1 --2- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff98808ecb0 0x7ff9880903d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7ff978009810 tx=0x7ff978009060).stop 2024-04-26T09:41:06.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.903+0000 7ff98f81a700 1 -- 172.21.15.194:0/1199606348 shutdown_connections 2024-04-26T09:41:06.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.903+0000 7ff98f81a700 1 --2- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff988090db0 0x7ff98812bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.903+0000 7ff98f81a700 1 --2- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff98808ecb0 0x7ff9880903d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.903+0000 7ff98f81a700 1 --2- 172.21.15.194:0/1199606348 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff98805e230 0x7ff9880908c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.903+0000 7ff98f81a700 1 -- 172.21.15.194:0/1199606348 >> 172.21.15.194:0/1199606348 conn(0x7ff9880914c0 msgr2=0x7ff9880918d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:06.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.903+0000 7ff98f81a700 1 -- 172.21.15.194:0/1199606348 shutdown_connections 2024-04-26T09:41:06.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.903+0000 7ff98f81a700 1 -- 172.21.15.194:0/1199606348 wait complete. 2024-04-26T09:41:06.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98f81a700 1 Processor -- start 2024-04-26T09:41:06.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98f81a700 1 -- start start 2024-04-26T09:41:06.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98f81a700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff98805e230 0x7ff98811a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98f81a700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff98808ecb0 0x7ff98811a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98f81a700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff988090db0 0x7ff98811ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:06.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98f81a700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98811dcb0 con 0x7ff98805e230 2024-04-26T09:41:06.913 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98f81a700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98811d2d0 con 0x7ff98808ecb0 2024-04-26T09:41:06.913 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98f81a700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff98812bf20 con 0x7ff988090db0 2024-04-26T09:41:06.913 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98d5b6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff98805e230 0x7ff98811a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.913 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98d5b6700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff98805e230 0x7ff98811a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44578/0 (socket says 172.21.15.194:44578) 2024-04-26T09:41:06.913 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98d5b6700 1 -- 172.21.15.194:0/2460203833 learned_addr learned my addr 172.21.15.194:0/2460203833 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:06.913 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98ddb7700 1 --2- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff988090db0 0x7ff98811ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.913 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.904+0000 7ff98cdb5700 1 --2- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff98808ecb0 0x7ff98811a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:06.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff98cdb5700 1 -- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff988090db0 msgr2=0x7ff98811ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff98cdb5700 1 --2- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff988090db0 0x7ff98811ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff98cdb5700 1 -- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff98805e230 msgr2=0x7ff98811a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:06.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff98cdb5700 1 --2- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff98805e230 0x7ff98811a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:06.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff98cdb5700 1 -- 172.21.15.194:0/2460203833 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9840092a0 con 0x7ff98808ecb0 2024-04-26T09:41:06.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff98d5b6700 1 --2- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff98805e230 0x7ff98811a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff98ddb7700 1 --2- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff988090db0 0x7ff98811ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:06.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff98cdb5700 1 --2- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff98808ecb0 0x7ff98811a840 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7ff98400e880 tx=0x7ff984009060).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:06.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff97dffb700 1 -- 172.21.15.194:0/2460203833 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9840094e0 con 0x7ff98808ecb0 2024-04-26T09:41:06.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff97dffb700 1 -- 172.21.15.194:0/2460203833 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984012600 con 0x7ff98808ecb0 2024-04-26T09:41:06.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff98f81a700 1 -- 172.21.15.194:0/2460203833 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff980009a20 con 0x7ff98808ecb0 2024-04-26T09:41:06.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff97dffb700 1 -- 172.21.15.194:0/2460203833 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff984012980 con 0x7ff98808ecb0 2024-04-26T09:41:06.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.905+0000 7ff98f81a700 1 -- 172.21.15.194:0/2460203833 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff978009300 con 0x7ff98808ecb0 2024-04-26T09:41:06.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.906+0000 7ff97dffb700 1 -- 172.21.15.194:0/2460203833 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7ff98401b400 con 0x7ff98808ecb0 2024-04-26T09:41:06.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.906+0000 7ff98f81a700 1 -- 172.21.15.194:0/2460203833 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff988091fe0 con 0x7ff98808ecb0 2024-04-26T09:41:06.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.906+0000 7ff97dffb700 1 -- 172.21.15.194:0/2460203833 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7ff9840555e0 con 0x7ff98808ecb0 2024-04-26T09:41:06.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:06.910+0000 7ff97dffb700 1 -- 172.21.15.194:0/2460203833 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7ff984036150 con 0x7ff98808ecb0 2024-04-26T09:41:07.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.051+0000 7ff98f81a700 1 -- 172.21.15.194:0/2460203833 --> v1:172.21.15.194:6816/72244 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff988046ce0 con 0x7ff96c041b80 2024-04-26T09:41:07.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.052+0000 7ff97dffb700 1 -- 172.21.15.194:0/2460203833 <== mgr.4111 v1:172.21.15.194:6816/72244 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (unknown 3526570807 0 0) 0x7ff988046ce0 con 0x7ff96c041b80 2024-04-26T09:41:07.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.053+0000 7ff98f81a700 1 -- 172.21.15.194:0/2460203833 >> v1:172.21.15.194:6816/72244 conn(0x7ff96c041b80 legacy=0x7ff96c044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:07.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.053+0000 7ff98f81a700 1 -- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff98808ecb0 msgr2=0x7ff98811a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:07.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.053+0000 7ff98f81a700 1 --2- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff98808ecb0 0x7ff98811a840 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7ff98400e880 tx=0x7ff984009060).stop 2024-04-26T09:41:07.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.053+0000 7ff98f81a700 1 -- 172.21.15.194:0/2460203833 shutdown_connections 2024-04-26T09:41:07.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.053+0000 7ff98f81a700 1 --2- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff988090db0 0x7ff98811ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:07.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.053+0000 7ff98f81a700 1 --2- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff98805e230 0x7ff98811a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:07.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.053+0000 7ff98f81a700 1 --2- 172.21.15.194:0/2460203833 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff98808ecb0 0x7ff98811a840 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:07.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.053+0000 7ff98f81a700 1 -- 172.21.15.194:0/2460203833 >> 172.21.15.194:0/2460203833 conn(0x7ff9880914c0 msgr2=0x7ff988092830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:07.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.053+0000 7ff98f81a700 1 -- 172.21.15.194:0/2460203833 shutdown_connections 2024-04-26T09:41:07.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.053+0000 7ff98f81a700 1 -- 172.21.15.194:0/2460203833 wait complete. 2024-04-26T09:41:07.058 INFO:teuthology.orchestra.run.smithi194.stderr:instructed osd(s) 3 to deep-scrub 2024-04-26T09:41:07.068 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T09:41:07.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.179+0000 7fd6eb3b7700 1 Processor -- start 2024-04-26T09:41:07.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6eb3b7700 1 -- start start 2024-04-26T09:41:07.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6eb3b7700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e4065c20 0x7fd6e4066000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:07.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6eb3b7700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fd6e41226a0 0x7fd6e412a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:07.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6eb3b7700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e40695c0 con 0x7fd6e40664f0 2024-04-26T09:41:07.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6eb3b7700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e4069810 con 0x7fd6e41226a0 2024-04-26T09:41:07.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6eb3b7700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4069290 con 0x7fd6e4065c20 2024-04-26T09:41:07.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6e8952700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd6e40664f0 0x7fd6e4121f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:52948/0 (socket says 172.21.15.194:52948) 2024-04-26T09:41:07.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6e9153700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e4065c20 0x7fd6e4066000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:07.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6e8952700 1 -- 172.21.15.194:0/3939963139 learned_addr learned my addr 172.21.15.194:0/3939963139 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:41:07.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6e9954700 1 --2- 172.21.15.194:0/3939963139 >> v2:172.21.15.194:3300/0 conn(0x7fd6e41226a0 0x7fd6e412a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:07.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6dbfff700 1 -- 172.21.15.194:0/3939963139 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651742317 0 0) 0x7fd6e40695c0 con 0x7fd6e40664f0 2024-04-26T09:41:07.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6dbfff700 1 -- 172.21.15.194:0/3939963139 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d00033f0 con 0x7fd6e40664f0 2024-04-26T09:41:07.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6e9153700 1 -- 172.21.15.194:0/3939963139 >> v2:172.21.15.194:3300/0 conn(0x7fd6e41226a0 msgr2=0x7fd6e412a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:07.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6e9153700 1 --2- 172.21.15.194:0/3939963139 >> v2:172.21.15.194:3300/0 conn(0x7fd6e41226a0 0x7fd6e412a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:07.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6e9153700 1 -- 172.21.15.194:0/3939963139 >> v1:172.21.15.194:6789/0 conn(0x7fd6e40664f0 legacy=0x7fd6e4121f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:07.193 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6e9153700 1 -- 172.21.15.194:0/3939963139 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e40684c0 con 0x7fd6e4065c20 2024-04-26T09:41:07.193 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.180+0000 7fd6e9954700 1 --2- 172.21.15.194:0/3939963139 >> v2:172.21.15.194:3300/0 conn(0x7fd6e41226a0 0x7fd6e412a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:07.193 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6e9153700 1 --2- 172.21.15.194:0/3939963139 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e4065c20 0x7fd6e4066000 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd6d4009760 tx=0x7fd6d4009060).ready entity=mon.2 client_cookie=bd5b46fc71b889e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:07.193 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6dbfff700 1 -- 172.21.15.194:0/3939963139 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d40090d0 con 0x7fd6e4065c20 2024-04-26T09:41:07.193 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6dbfff700 1 -- 172.21.15.194:0/3939963139 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d400d4a0 con 0x7fd6e4065c20 2024-04-26T09:41:07.193 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6dbfff700 1 -- 172.21.15.194:0/3939963139 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d400d820 con 0x7fd6e4065c20 2024-04-26T09:41:07.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3939963139 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e4065c20 msgr2=0x7fd6e4066000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:07.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6eb3b7700 1 --2- 172.21.15.194:0/3939963139 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e4065c20 0x7fd6e4066000 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd6d4009760 tx=0x7fd6d4009060).stop 2024-04-26T09:41:07.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3939963139 shutdown_connections 2024-04-26T09:41:07.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6eb3b7700 1 --2- 172.21.15.194:0/3939963139 >> v2:172.21.15.194:3300/0 conn(0x7fd6e41226a0 0x7fd6e412a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:07.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6eb3b7700 1 --2- 172.21.15.194:0/3939963139 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e4065c20 0x7fd6e4066000 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:07.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3939963139 >> 172.21.15.194:0/3939963139 conn(0x7fd6e40957b0 msgr2=0x7fd6e4095bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:07.195 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3939963139 shutdown_connections 2024-04-26T09:41:07.195 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.181+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3939963139 wait complete. 2024-04-26T09:41:07.195 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.182+0000 7fd6eb3b7700 1 Processor -- start 2024-04-26T09:41:07.195 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.182+0000 7fd6eb3b7700 1 -- start start 2024-04-26T09:41:07.195 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6eb3b7700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd6e4065c20 0x7fd6e4112fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:07.195 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6eb3b7700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e40664f0 0x7fd6e41134c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:07.195 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6eb3b7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6e41226a0 0x7fd6e41139b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:07.196 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6eb3b7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e41164f0 con 0x7fd6e41226a0 2024-04-26T09:41:07.196 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6eb3b7700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4114be0 con 0x7fd6e4065c20 2024-04-26T09:41:07.196 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6eb3b7700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4113ea0 con 0x7fd6e40664f0 2024-04-26T09:41:07.196 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e8952700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e40664f0 0x7fd6e41134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:07.196 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e9954700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6e41226a0 0x7fd6e41139b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:07.196 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e9954700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6e41226a0 0x7fd6e41139b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44588/0 (socket says 172.21.15.194:44588) 2024-04-26T09:41:07.197 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e8952700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e40664f0 0x7fd6e41134c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3302/0 says I am v2:172.21.15.194:47872/0 (socket says 172.21.15.194:47872) 2024-04-26T09:41:07.197 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e9954700 1 -- 172.21.15.194:0/3274045549 learned_addr learned my addr 172.21.15.194:0/3274045549 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:07.197 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e9153700 1 --2- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd6e4065c20 0x7fd6e4112fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:07.197 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e9954700 1 -- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e40664f0 msgr2=0x7fd6e41134c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:07.197 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e9954700 1 --2- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e40664f0 0x7fd6e41134c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:07.197 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e9954700 1 -- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd6e4065c20 msgr2=0x7fd6e4112fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:07.197 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e9954700 1 --2- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd6e4065c20 0x7fd6e4112fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:07.197 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e9954700 1 -- 172.21.15.194:0/3274045549 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc009a20 con 0x7fd6e41226a0 2024-04-26T09:41:07.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e8952700 1 --2- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e40664f0 0x7fd6e41134c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:07.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e9153700 1 --2- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd6e4065c20 0x7fd6e4112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:07.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6e9954700 1 --2- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6e41226a0 0x7fd6e41139b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd6dc009fc0 tx=0x7fd6dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:07.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6d9ffb700 1 -- 172.21.15.194:0/3274045549 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc009d20 con 0x7fd6e41226a0 2024-04-26T09:41:07.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6d9ffb700 1 -- 172.21.15.194:0/3274045549 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6dc00f040 con 0x7fd6e41226a0 2024-04-26T09:41:07.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3274045549 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d40092a0 con 0x7fd6e41226a0 2024-04-26T09:41:07.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.183+0000 7fd6d9ffb700 1 -- 172.21.15.194:0/3274045549 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc00cde0 con 0x7fd6e41226a0 2024-04-26T09:41:07.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.184+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3274045549 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e4115ed0 con 0x7fd6e41226a0 2024-04-26T09:41:07.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.184+0000 7fd6d9ffb700 1 -- 172.21.15.194:0/3274045549 <== mon.0 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fd6dc013610 con 0x7fd6e41226a0 2024-04-26T09:41:07.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.185+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3274045549 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e4092a10 con 0x7fd6e41226a0 2024-04-26T09:41:07.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.185+0000 7fd6d9ffb700 1 -- 172.21.15.194:0/3274045549 <== mon.0 v2:172.21.15.194:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fd6dc055ad0 con 0x7fd6e41226a0 2024-04-26T09:41:07.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.188+0000 7fd6d9ffb700 1 -- 172.21.15.194:0/3274045549 <== mon.0 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fd6dc036640 con 0x7fd6e41226a0 2024-04-26T09:41:07.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.324+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3274045549 --> v1:172.21.15.194:6816/72244 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd6e4046ce0 con 0x7fd6c8041b80 2024-04-26T09:41:07.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.326+0000 7fd6d9ffb700 1 -- 172.21.15.194:0/3274045549 <== mgr.4111 v1:172.21.15.194:6816/72244 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+37941 (unknown 2965378022 0 3911581024) 0x7fd6e4046ce0 con 0x7fd6c8041b80 2024-04-26T09:41:07.329 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:41:07.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.327+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3274045549 >> v1:172.21.15.194:6816/72244 conn(0x7fd6c8041b80 legacy=0x7fd6c8044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:07.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.327+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6e41226a0 msgr2=0x7fd6e41139b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:07.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.327+0000 7fd6eb3b7700 1 --2- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6e41226a0 0x7fd6e41139b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd6dc009fc0 tx=0x7fd6dc00b040).stop 2024-04-26T09:41:07.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.327+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3274045549 shutdown_connections 2024-04-26T09:41:07.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.327+0000 7fd6eb3b7700 1 --2- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6e41226a0 0x7fd6e41139b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:07.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.327+0000 7fd6eb3b7700 1 --2- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fd6e4065c20 0x7fd6e4112fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:07.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.327+0000 7fd6eb3b7700 1 --2- 172.21.15.194:0/3274045549 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fd6e40664f0 0x7fd6e41134c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:07.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.327+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3274045549 >> 172.21.15.194:0/3274045549 conn(0x7fd6e40957b0 msgr2=0x7fd6e41208f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:07.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.327+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3274045549 shutdown_connections 2024-04-26T09:41:07.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:07.327+0000 7fd6eb3b7700 1 -- 172.21.15.194:0/3274045549 wait complete. 2024-04-26T09:41:07.331 INFO:teuthology.orchestra.run.smithi194.stderr:dumped all 2024-04-26T09:41:07.341 INFO:teuthology.orchestra.run.smithi194.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-04-26T09:41:06.499582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1188832,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":1188352,"kb_avail":376298528,"statfs":{"total":386547056640,"available":385329692672,"internally_reserved":0,"allocated":458752,"data_stored":122196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1216872448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787045+0000","last_change":"2024-04-26T09:35:39.476064+0000","last_active":"2024-04-26T09:35:42.787045+0000","last_peered":"2024-04-26T09:35:42.787045+0000","last_clean":"2024-04-26T09:35:42.787045+0000","last_became_active":"2024-04-26T09:35:39.475797+0000","last_became_peered":"2024-04-26T09:35:39.475797+0000","last_unstale":"2024-04-26T09:35:42.787045+0000","last_undegraded":"2024-04-26T09:35:42.787045+0000","last_fullsized":"2024-04-26T09:35:42.787045+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.477037+0000","last_change":"2024-04-26T09:35:39.482000+0000","last_active":"2024-04-26T09:35:40.477037+0000","last_peered":"2024-04-26T09:35:40.477037+0000","last_clean":"2024-04-26T09:35:40.477037+0000","last_became_active":"2024-04-26T09:35:39.481746+0000","last_became_peered":"2024-04-26T09:35:39.481746+0000","last_unstale":"2024-04-26T09:35:40.477037+0000","last_undegraded":"2024-04-26T09:35:40.477037+0000","last_fullsized":"2024-04-26T09:35:40.477037+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787082+0000","last_change":"2024-04-26T09:35:39.473129+0000","last_active":"2024-04-26T09:35:42.787082+0000","last_peered":"2024-04-26T09:35:42.787082+0000","last_clean":"2024-04-26T09:35:42.787082+0000","last_became_active":"2024-04-26T09:35:39.472876+0000","last_became_peered":"2024-04-26T09:35:39.472876+0000","last_unstale":"2024-04-26T09:35:42.787082+0000","last_undegraded":"2024-04-26T09:35:42.787082+0000","last_fullsized":"2024-04-26T09:35:42.787082+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787021+0000","last_change":"2024-04-26T09:35:39.482206+0000","last_active":"2024-04-26T09:35:42.787021+0000","last_peered":"2024-04-26T09:35:42.787021+0000","last_clean":"2024-04-26T09:35:42.787021+0000","last_became_active":"2024-04-26T09:35:39.481898+0000","last_became_peered":"2024-04-26T09:35:39.481898+0000","last_unstale":"2024-04-26T09:35:42.787021+0000","last_undegraded":"2024-04-26T09:35:42.787021+0000","last_fullsized":"2024-04-26T09:35:42.787021+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.476890+0000","last_change":"2024-04-26T09:35:39.481874+0000","last_active":"2024-04-26T09:35:40.476890+0000","last_peered":"2024-04-26T09:35:40.476890+0000","last_clean":"2024-04-26T09:35:40.476890+0000","last_became_active":"2024-04-26T09:35:39.481585+0000","last_became_peered":"2024-04-26T09:35:39.481585+0000","last_unstale":"2024-04-26T09:35:40.476890+0000","last_undegraded":"2024-04-26T09:35:40.476890+0000","last_fullsized":"2024-04-26T09:35:40.476890+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787076+0000","last_change":"2024-04-26T09:35:39.478633+0000","last_active":"2024-04-26T09:35:42.787076+0000","last_peered":"2024-04-26T09:35:42.787076+0000","last_clean":"2024-04-26T09:35:42.787076+0000","last_became_active":"2024-04-26T09:35:39.478349+0000","last_became_peered":"2024-04-26T09:35:39.478349+0000","last_unstale":"2024-04-26T09:35:42.787076+0000","last_undegraded":"2024-04-26T09:35:42.787076+0000","last_fullsized":"2024-04-26T09:35:42.787076+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.786959+0000","last_change":"2024-04-26T09:35:39.481494+0000","last_active":"2024-04-26T09:35:42.786959+0000","last_peered":"2024-04-26T09:35:42.786959+0000","last_clean":"2024-04-26T09:35:42.786959+0000","last_became_active":"2024-04-26T09:35:39.481144+0000","last_became_peered":"2024-04-26T09:35:39.481144+0000","last_unstale":"2024-04-26T09:35:42.786959+0000","last_undegraded":"2024-04-26T09:35:42.786959+0000","last_fullsized":"2024-04-26T09:35:42.786959+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:42.787040+0000","last_change":"2024-04-26T09:35:39.481502+0000","last_active":"2024-04-26T09:35:42.787040+0000","last_peered":"2024-04-26T09:35:42.787040+0000","last_clean":"2024-04-26T09:35:42.787040+0000","last_became_active":"2024-04-26T09:35:39.481116+0000","last_became_peered":"2024-04-26T09:35:39.481116+0000","last_unstale":"2024-04-26T09:35:42.787040+0000","last_undegraded":"2024-04-26T09:35:42.787040+0000","last_fullsized":"2024-04-26T09:35:42.787040+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-26T09:35:40.476945+0000","last_change":"2024-04-26T09:35:39.481452+0000","last_active":"2024-04-26T09:35:40.476945+0000","last_peered":"2024-04-26T09:35:40.476945+0000","last_clean":"2024-04-26T09:35:40.476945+0000","last_became_active":"2024-04-26T09:35:39.481155+0000","last_became_peered":"2024-04-26T09:35:39.481155+0000","last_unstale":"2024-04-26T09:35:40.476945+0000","last_undegraded":"2024-04-26T09:35:40.476945+0000","last_fullsized":"2024-04-26T09:35:40.476945+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:35:38.454450+0000","last_clean_scrub_stamp":"2024-04-26T09:35:38.454450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.34899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.51000000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40300000000000002,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.28199999999999997}]},{"osd":1,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35999999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47799999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.36799999999999999,"15min":0.374},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.40200000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.49299999999999999,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.57299999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.312}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.22600000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.45900000000000002,"5min":0.498,"15min":0.498},"last":0.247},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.51500000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.28899999999999998}]},{"osd":1,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.35399999999999998,"15min":0.35599999999999998},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.45900000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.253}]},{"osd":3,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38800000000000001,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48199999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.499,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.28199999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705732,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.33800000000000002,"15min":0.33700000000000002},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":0.47199999999999998,"5min":0.504,"15min":0.504},"last":0.20699999999999999},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34200000000000003,"15min":0.34100000000000003},"min":{"1min":0.186,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47299999999999998,"5min":0.504,"15min":0.504},"last":0.224}]},{"osd":2,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.46999999999999997,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.251},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.21099999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.23799999999999999}]},{"osd":3,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51900000000000002,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51300000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.28499999999999998}]}]},{"osd":1,"up_from":9,"seq":38654705731,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.311},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.39000000000000001},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.30499999999999999}]},{"osd":2,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"min":{"1min":0.245,"5min":0.185,"15min":0.185},"max":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.35399999999999998},"min":{"1min":0.26000000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.34799999999999998}]},{"osd":3,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.36699999999999999,"15min":0.34399999999999997},"min":{"1min":0.25900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"min":{"1min":0.27600000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55500000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T09:41:07.344 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-26T09:35:38.454450+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=35, tm_sec=38, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=41, tm_sec=4, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T09:41:07.344 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-26T09:35:38.454450+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=35, tm_sec=38, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=41, tm_sec=4, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T09:41:07.345 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-26T09:35:38.454450+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=35, tm_sec=38, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=41, tm_sec=4, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T09:41:07.345 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-26T09:35:38.454450+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=35, tm_sec=38, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=41, tm_sec=4, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T09:41:07.345 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-26T09:35:38.454450+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=35, tm_sec=38, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=41, tm_sec=4, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T09:41:07.345 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-26T09:35:38.454450+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=35, tm_sec=38, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=41, tm_sec=4, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T09:41:07.345 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-26T09:35:38.454450+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=35, tm_sec=38, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=41, tm_sec=4, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T09:41:07.345 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-26T09:35:38.454450+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=35, tm_sec=38, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=41, tm_sec=4, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T09:41:07.346 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-26T09:35:38.454450+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=35, tm_sec=38, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=9, tm_min=41, tm_sec=4, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T09:41:07.346 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-26T09:41:27.347 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T09:41:27.475 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.472+0000 7ff223db9700 1 Processor -- start 2024-04-26T09:41:27.475 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.472+0000 7ff223db9700 1 -- start start 2024-04-26T09:41:27.479 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff223db9700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7ff21c05e230 0x7ff21c0654a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:27.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff223db9700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c12acb0 0x7ff21c12d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:27.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff223db9700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21c06a6c0 con 0x7ff21c065990 2024-04-26T09:41:27.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff223db9700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff21c05e8c0 con 0x7ff21c05e230 2024-04-26T09:41:27.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff223db9700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c0665c0 con 0x7ff21c12acb0 2024-04-26T09:41:27.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff222356700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c12acb0 0x7ff21c12d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:27.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff221b55700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7ff21c05e230 0x7ff21c0654a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:27.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff221b55700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7ff21c05e230 0x7ff21c0654a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48868/0 (socket says 172.21.15.194:48868) 2024-04-26T09:41:27.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff222356700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c12acb0 0x7ff21c12d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:40530/0 (socket says 172.21.15.194:40530) 2024-04-26T09:41:27.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff221b55700 1 -- 172.21.15.194:0/2010502136 learned_addr learned my addr 172.21.15.194:0/2010502136 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:27.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff221354700 1 --1- 172.21.15.194:0/2010502136 >> v1:172.21.15.194:6789/0 conn(0x7ff21c065990 0x7ff21c065d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:37796/0 (socket says 172.21.15.194:37796) 2024-04-26T09:41:27.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff221b55700 1 -- 172.21.15.194:0/2010502136 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c12acb0 msgr2=0x7ff21c12d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:27.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff221b55700 1 --2- 172.21.15.194:0/2010502136 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c12acb0 0x7ff21c12d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff221b55700 1 -- 172.21.15.194:0/2010502136 >> v1:172.21.15.194:6789/0 conn(0x7ff21c065990 legacy=0x7ff21c065d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff220b53700 1 -- 172.21.15.194:0/2010502136 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314094597 0 0) 0x7ff21c06a6c0 con 0x7ff21c065990 2024-04-26T09:41:27.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff221b55700 1 -- 172.21.15.194:0/2010502136 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c08eec0 con 0x7ff21c05e230 2024-04-26T09:41:27.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.473+0000 7ff222356700 1 --2- 172.21.15.194:0/2010502136 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c12acb0 0x7ff21c12d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:27.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff221b55700 1 --2- 172.21.15.194:0/2010502136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff21c05e230 0x7ff21c0654a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7ff218009810 tx=0x7ff218009060).ready entity=mon.0 client_cookie=7f0716b858aec560 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:27.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff220b53700 1 -- 172.21.15.194:0/2010502136 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218005880 con 0x7ff21c05e230 2024-04-26T09:41:27.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff220b53700 1 -- 172.21.15.194:0/2010502136 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2180059e0 con 0x7ff21c05e230 2024-04-26T09:41:27.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff220b53700 1 -- 172.21.15.194:0/2010502136 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218005d60 con 0x7ff21c05e230 2024-04-26T09:41:27.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff223db9700 1 -- 172.21.15.194:0/2010502136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff21c05e230 msgr2=0x7ff21c0654a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff223db9700 1 --2- 172.21.15.194:0/2010502136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff21c05e230 0x7ff21c0654a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7ff218009810 tx=0x7ff218009060).stop 2024-04-26T09:41:27.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff223db9700 1 -- 172.21.15.194:0/2010502136 shutdown_connections 2024-04-26T09:41:27.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff223db9700 1 --2- 172.21.15.194:0/2010502136 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c12acb0 0x7ff21c12d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff223db9700 1 --2- 172.21.15.194:0/2010502136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff21c05e230 0x7ff21c0654a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff223db9700 1 -- 172.21.15.194:0/2010502136 >> 172.21.15.194:0/2010502136 conn(0x7ff21c091910 msgr2=0x7ff21c091d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:27.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff223db9700 1 -- 172.21.15.194:0/2010502136 shutdown_connections 2024-04-26T09:41:27.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.474+0000 7ff223db9700 1 -- 172.21.15.194:0/2010502136 wait complete. 2024-04-26T09:41:27.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.475+0000 7ff223db9700 1 Processor -- start 2024-04-26T09:41:27.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.475+0000 7ff223db9700 1 -- start start 2024-04-26T09:41:27.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.475+0000 7ff223db9700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c05e230 0x7ff21c1290f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:27.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.475+0000 7ff223db9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff21c065990 0x7ff21c1295e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:27.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.475+0000 7ff223db9700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff21c12acb0 0x7ff21c129ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:27.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff223db9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c088890 con 0x7ff21c065990 2024-04-26T09:41:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff223db9700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c12c7d0 con 0x7ff21c05e230 2024-04-26T09:41:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff223db9700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c088e30 con 0x7ff21c12acb0 2024-04-26T09:41:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221354700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff21c065990 0x7ff21c1295e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221b55700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c05e230 0x7ff21c1290f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221b55700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c05e230 0x7ff21c1290f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3301/0 says I am v2:172.21.15.194:40532/0 (socket says 172.21.15.194:40532) 2024-04-26T09:41:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221354700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff21c065990 0x7ff21c1295e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48876/0 (socket says 172.21.15.194:48876) 2024-04-26T09:41:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221b55700 1 -- 172.21.15.194:0/1529975981 learned_addr learned my addr 172.21.15.194:0/1529975981 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff222356700 1 --2- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff21c12acb0 0x7ff21c129ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:27.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221b55700 1 -- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff21c12acb0 msgr2=0x7ff21c129ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221b55700 1 --2- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff21c12acb0 0x7ff21c129ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221b55700 1 -- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff21c065990 msgr2=0x7ff21c1295e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221b55700 1 --2- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff21c065990 0x7ff21c1295e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221b55700 1 -- 172.21.15.194:0/1529975981 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff214009a20 con 0x7ff21c05e230 2024-04-26T09:41:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221354700 1 --2- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff21c065990 0x7ff21c1295e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff221b55700 1 --2- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c05e230 0x7ff21c1290f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff218009eb0 tx=0x7ff218016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:27.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff20e7fc700 1 -- 172.21.15.194:0/1529975981 <== mon.1 v2:172.21.15.194:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218005880 con 0x7ff21c05e230 2024-04-26T09:41:27.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff20e7fc700 1 -- 172.21.15.194:0/1529975981 <== mon.1 v2:172.21.15.194:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21800b790 con 0x7ff21c05e230 2024-04-26T09:41:27.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff223db9700 1 -- 172.21.15.194:0/1529975981 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff218009300 con 0x7ff21c05e230 2024-04-26T09:41:27.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff20e7fc700 1 -- 172.21.15.194:0/1529975981 <== mon.1 v2:172.21.15.194:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21800bb30 con 0x7ff21c05e230 2024-04-26T09:41:27.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.476+0000 7ff223db9700 1 -- 172.21.15.194:0/1529975981 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21c08df60 con 0x7ff21c05e230 2024-04-26T09:41:27.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.477+0000 7ff20e7fc700 1 -- 172.21.15.194:0/1529975981 <== mon.1 v2:172.21.15.194:3301/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7ff218018070 con 0x7ff21c05e230 2024-04-26T09:41:27.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.477+0000 7ff223db9700 1 -- 172.21.15.194:0/1529975981 --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21c0891e0 con 0x7ff21c05e230 2024-04-26T09:41:27.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.477+0000 7ff20e7fc700 1 -- 172.21.15.194:0/1529975981 <== mon.1 v2:172.21.15.194:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7ff218056120 con 0x7ff21c05e230 2024-04-26T09:41:27.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.481+0000 7ff20e7fc700 1 -- 172.21.15.194:0/1529975981 <== mon.1 v2:172.21.15.194:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7ff2180077a0 con 0x7ff21c05e230 2024-04-26T09:41:27.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.618+0000 7ff223db9700 1 -- 172.21.15.194:0/1529975981 --> v1:172.21.15.194:6816/72244 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff21c0885b0 con 0x7ff200041b80 2024-04-26T09:41:27.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.619+0000 7ff20e7fc700 1 -- 172.21.15.194:0/1529975981 <== mgr.4111 v1:172.21.15.194:6816/72244 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+38007 (unknown 2965378022 0 586077148) 0x7ff21c0885b0 con 0x7ff200041b80 2024-04-26T09:41:27.622 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-26T09:41:27.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.620+0000 7ff223db9700 1 -- 172.21.15.194:0/1529975981 >> v1:172.21.15.194:6816/72244 conn(0x7ff200041b80 legacy=0x7ff200044040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.620+0000 7ff223db9700 1 -- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c05e230 msgr2=0x7ff21c1290f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.620+0000 7ff223db9700 1 --2- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c05e230 0x7ff21c1290f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7ff218009eb0 tx=0x7ff218016000).stop 2024-04-26T09:41:27.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.621+0000 7ff223db9700 1 -- 172.21.15.194:0/1529975981 shutdown_connections 2024-04-26T09:41:27.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.621+0000 7ff223db9700 1 --2- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7ff21c12acb0 0x7ff21c129ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.621+0000 7ff223db9700 1 --2- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7ff21c05e230 0x7ff21c1290f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.621+0000 7ff223db9700 1 --2- 172.21.15.194:0/1529975981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff21c065990 0x7ff21c1295e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.621+0000 7ff223db9700 1 -- 172.21.15.194:0/1529975981 >> 172.21.15.194:0/1529975981 conn(0x7ff21c091910 msgr2=0x7ff21c093730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:27.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.621+0000 7ff223db9700 1 -- 172.21.15.194:0/1529975981 shutdown_connections 2024-04-26T09:41:27.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.621+0000 7ff223db9700 1 -- 172.21.15.194:0/1529975981 wait complete. 2024-04-26T09:41:27.625 INFO:teuthology.orchestra.run.smithi194.stderr:dumped all 2024-04-26T09:41:27.634 INFO:teuthology.orchestra.run.smithi194.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-04-26T09:41:26.503605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1188832,"kb_used_data":448,"kb_used_omap":0,"kb_used_meta":1188352,"kb_avail":376298528,"statfs":{"total":386547056640,"available":385329692672,"internally_reserved":0,"allocated":458752,"data_stored":122196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1216872448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-26T09:41:23.444465+0000","last_change":"2024-04-26T09:41:23.444465+0000","last_active":"2024-04-26T09:41:23.444465+0000","last_peered":"2024-04-26T09:41:23.444465+0000","last_clean":"2024-04-26T09:41:23.444465+0000","last_became_active":"2024-04-26T09:35:39.475797+0000","last_became_peered":"2024-04-26T09:35:39.475797+0000","last_unstale":"2024-04-26T09:41:23.444465+0000","last_undegraded":"2024-04-26T09:41:23.444465+0000","last_fullsized":"2024-04-26T09:41:23.444465+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:41:23.444319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:41:23.444319+0000","last_clean_scrub_stamp":"2024-04-26T09:41:23.444319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-26T09:41:14.199246+0000","last_change":"2024-04-26T09:41:14.199246+0000","last_active":"2024-04-26T09:41:14.199246+0000","last_peered":"2024-04-26T09:41:14.199246+0000","last_clean":"2024-04-26T09:41:14.199246+0000","last_became_active":"2024-04-26T09:35:39.481746+0000","last_became_peered":"2024-04-26T09:35:39.481746+0000","last_unstale":"2024-04-26T09:41:14.199246+0000","last_undegraded":"2024-04-26T09:41:14.199246+0000","last_fullsized":"2024-04-26T09:41:14.199246+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:41:14.199110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:41:14.199110+0000","last_clean_scrub_stamp":"2024-04-26T09:41:14.199110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-26T09:41:20.397624+0000","last_change":"2024-04-26T09:41:20.397624+0000","last_active":"2024-04-26T09:41:20.397624+0000","last_peered":"2024-04-26T09:41:20.397624+0000","last_clean":"2024-04-26T09:41:20.397624+0000","last_became_active":"2024-04-26T09:35:39.472876+0000","last_became_peered":"2024-04-26T09:35:39.472876+0000","last_unstale":"2024-04-26T09:41:20.397624+0000","last_undegraded":"2024-04-26T09:41:20.397624+0000","last_fullsized":"2024-04-26T09:41:20.397624+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:41:20.397479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:41:20.397479+0000","last_clean_scrub_stamp":"2024-04-26T09:41:20.397479+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-26T09:41:13.546432+0000","last_change":"2024-04-26T09:41:13.546432+0000","last_active":"2024-04-26T09:41:13.546432+0000","last_peered":"2024-04-26T09:41:13.546432+0000","last_clean":"2024-04-26T09:41:13.546432+0000","last_became_active":"2024-04-26T09:35:39.481898+0000","last_became_peered":"2024-04-26T09:35:39.481898+0000","last_unstale":"2024-04-26T09:41:13.546432+0000","last_undegraded":"2024-04-26T09:41:13.546432+0000","last_fullsized":"2024-04-26T09:41:13.546432+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:41:13.546320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:41:13.546320+0000","last_clean_scrub_stamp":"2024-04-26T09:41:13.546320+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-26T09:41:09.179450+0000","last_change":"2024-04-26T09:41:09.179450+0000","last_active":"2024-04-26T09:41:09.179450+0000","last_peered":"2024-04-26T09:41:09.179450+0000","last_clean":"2024-04-26T09:41:09.179450+0000","last_became_active":"2024-04-26T09:35:39.481585+0000","last_became_peered":"2024-04-26T09:35:39.481585+0000","last_unstale":"2024-04-26T09:41:09.179450+0000","last_undegraded":"2024-04-26T09:41:09.179450+0000","last_fullsized":"2024-04-26T09:41:09.179450+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:41:09.179342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:41:09.179342+0000","last_clean_scrub_stamp":"2024-04-26T09:41:09.179342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-26T09:41:17.458403+0000","last_change":"2024-04-26T09:41:17.458403+0000","last_active":"2024-04-26T09:41:17.458403+0000","last_peered":"2024-04-26T09:41:17.458403+0000","last_clean":"2024-04-26T09:41:17.458403+0000","last_became_active":"2024-04-26T09:35:39.478349+0000","last_became_peered":"2024-04-26T09:35:39.478349+0000","last_unstale":"2024-04-26T09:41:17.458403+0000","last_undegraded":"2024-04-26T09:41:17.458403+0000","last_fullsized":"2024-04-26T09:41:17.458403+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:41:17.458287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:41:17.458287+0000","last_clean_scrub_stamp":"2024-04-26T09:41:17.458287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-26T09:41:13.177782+0000","last_change":"2024-04-26T09:41:13.177782+0000","last_active":"2024-04-26T09:41:13.177782+0000","last_peered":"2024-04-26T09:41:13.177782+0000","last_clean":"2024-04-26T09:41:13.177782+0000","last_became_active":"2024-04-26T09:35:39.481144+0000","last_became_peered":"2024-04-26T09:35:39.481144+0000","last_unstale":"2024-04-26T09:41:13.177782+0000","last_undegraded":"2024-04-26T09:41:13.177782+0000","last_fullsized":"2024-04-26T09:41:13.177782+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:41:13.177626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:41:13.177626+0000","last_clean_scrub_stamp":"2024-04-26T09:41:13.177626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-26T09:41:15.503834+0000","last_change":"2024-04-26T09:41:15.503834+0000","last_active":"2024-04-26T09:41:15.503834+0000","last_peered":"2024-04-26T09:41:15.503834+0000","last_clean":"2024-04-26T09:41:15.503834+0000","last_became_active":"2024-04-26T09:35:39.481116+0000","last_became_peered":"2024-04-26T09:35:39.481116+0000","last_unstale":"2024-04-26T09:41:15.503834+0000","last_undegraded":"2024-04-26T09:41:15.503834+0000","last_fullsized":"2024-04-26T09:41:15.503834+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:41:15.503705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:41:15.503705+0000","last_clean_scrub_stamp":"2024-04-26T09:41:15.503705+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-26T09:41:12.166385+0000","last_change":"2024-04-26T09:41:12.166385+0000","last_active":"2024-04-26T09:41:12.166385+0000","last_peered":"2024-04-26T09:41:12.166385+0000","last_clean":"2024-04-26T09:41:12.166385+0000","last_became_active":"2024-04-26T09:35:39.481155+0000","last_became_peered":"2024-04-26T09:35:39.481155+0000","last_unstale":"2024-04-26T09:41:12.166385+0000","last_undegraded":"2024-04-26T09:41:12.166385+0000","last_fullsized":"2024-04-26T09:41:12.166385+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T09:41:12.166260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T09:41:12.166260+0000","last_clean_scrub_stamp":"2024-04-26T09:41:12.166260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.34499999999999997,"15min":0.34899999999999998},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.51000000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40300000000000002,"15min":0.40799999999999997},"min":{"1min":0.309,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.41199999999999998}]},{"osd":1,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35999999999999999,"15min":0.36399999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.47799999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"last":0.374},{"interface":"front","average":{"1min":0.373,"5min":0.36799999999999999,"15min":0.374},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.53100000000000003,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.38700000000000001}]},{"osd":2,"last update":"Fri Apr 26 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.40200000000000002},"min":{"1min":0.28999999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.49299999999999999,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.57299999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.47999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.22600000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.45900000000000002,"5min":0.498,"15min":0.498},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.51500000000000001,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.35399999999999998,"15min":0.35599999999999998},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.379},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.35899999999999999,"15min":0.36499999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.45900000000000002,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.32100000000000001}]},{"osd":3,"last update":"Fri Apr 26 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38800000000000001,"15min":0.38900000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.48199999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.499,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.36499999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705736,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.33800000000000002,"15min":0.33700000000000002},"min":{"1min":0.14399999999999999,"5min":0.14399999999999999,"15min":0.14399999999999999},"max":{"1min":0.47199999999999998,"5min":0.504,"15min":0.504},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.34200000000000003,"15min":0.34100000000000003},"min":{"1min":0.186,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.47299999999999998,"5min":0.504,"15min":0.504},"last":0.41799999999999998}]},{"osd":2,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.46999999999999997,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.36199999999999999,"15min":0.36599999999999999},"min":{"1min":0.21099999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.44800000000000001}]},{"osd":3,"last update":"Fri Apr 26 09:40:54 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51900000000000002,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.39100000000000001,"15min":0.39600000000000002},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.51300000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.48699999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705735,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297208,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074632,"statfs":{"total":96636764160,"available":96332423168,"internally_reserved":0,"allocated":114688,"data_stored":30549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.36199999999999999,"15min":0.34599999999999997},"min":{"1min":0.193,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40500000000000003,"15min":0.39000000000000001},"min":{"1min":0.28599999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.38400000000000001}]},{"osd":2,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.35899999999999999,"15min":0.33900000000000002},"min":{"1min":0.245,"5min":0.185,"15min":0.185},"max":{"1min":0.51200000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.36799999999999999,"15min":0.35399999999999998},"min":{"1min":0.26000000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.34699999999999998}]},{"osd":3,"last update":"Fri Apr 26 09:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.36699999999999999,"15min":0.34399999999999997},"min":{"1min":0.25900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.435},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39800000000000002,"15min":0.38300000000000001},"min":{"1min":0.27600000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.55500000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.35899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T09:41:27.636 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-26T09:41:27.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.746+0000 7fde48597700 1 Processor -- start 2024-04-26T09:41:27.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.746+0000 7fde48597700 1 -- start start 2024-04-26T09:41:27.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde48597700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065f20 0x7fde40126d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:27.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde48597700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fde401271f0 0x7fde401295e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:27.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde48597700 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde40091420 con 0x7fde40065430 2024-04-26T09:41:27.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde48597700 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde40066300 con 0x7fde401271f0 2024-04-26T09:41:27.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde48597700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde40067330 con 0x7fde40065f20 2024-04-26T09:41:27.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde46b34700 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fde401271f0 0x7fde401295e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:27.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde45b32700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065f20 0x7fde40126d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:27.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde46333700 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fde40065430 0x7fde40065810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:37802/0 (socket says 172.21.15.194:37802) 2024-04-26T09:41:27.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde46333700 1 -- 172.21.15.194:0/2621320632 learned_addr learned my addr 172.21.15.194:0/2621320632 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-04-26T09:41:27.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde46b34700 1 -- 172.21.15.194:0/2621320632 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065f20 msgr2=0x7fde40126d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T09:41:27.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde46b34700 1 --2- 172.21.15.194:0/2621320632 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065f20 0x7fde40126d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde46b34700 1 -- 172.21.15.194:0/2621320632 >> v1:172.21.15.194:6789/0 conn(0x7fde40065430 legacy=0x7fde40065810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde46b34700 1 -- 172.21.15.194:0/2621320632 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde400911e0 con 0x7fde401271f0 2024-04-26T09:41:27.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde45331700 1 -- 172.21.15.194:0/2621320632 <== mon.0 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092053499 0 0) 0x7fde40091420 con 0x7fde40065430 2024-04-26T09:41:27.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde45b32700 1 --2- 172.21.15.194:0/2621320632 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065f20 0x7fde40126d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:27.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde46b34700 1 --2- 172.21.15.194:0/2621320632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde401271f0 0x7fde401295e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fde38009e60 tx=0x7fde38009780).ready entity=mon.0 client_cookie=994c191540d7a981 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:27.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.747+0000 7fde45331700 1 -- 172.21.15.194:0/2621320632 <== mon.0 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3800e070 con 0x7fde401271f0 2024-04-26T09:41:27.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.748+0000 7fde45331700 1 -- 172.21.15.194:0/2621320632 <== mon.0 v2:172.21.15.194:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3800bb60 con 0x7fde401271f0 2024-04-26T09:41:27.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.748+0000 7fde45331700 1 -- 172.21.15.194:0/2621320632 <== mon.0 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38012410 con 0x7fde401271f0 2024-04-26T09:41:27.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.748+0000 7fde48597700 1 -- 172.21.15.194:0/2621320632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde401271f0 msgr2=0x7fde401295e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.748+0000 7fde48597700 1 --2- 172.21.15.194:0/2621320632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde401271f0 0x7fde401295e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fde38009e60 tx=0x7fde38009780).stop 2024-04-26T09:41:27.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.748+0000 7fde48597700 1 -- 172.21.15.194:0/2621320632 shutdown_connections 2024-04-26T09:41:27.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.748+0000 7fde48597700 1 --2- 172.21.15.194:0/2621320632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde401271f0 0x7fde401295e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.748+0000 7fde48597700 1 --2- 172.21.15.194:0/2621320632 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065f20 0x7fde40126d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.748+0000 7fde48597700 1 -- 172.21.15.194:0/2621320632 >> 172.21.15.194:0/2621320632 conn(0x7fde40094ce0 msgr2=0x7fde400950f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:27.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.748+0000 7fde48597700 1 -- 172.21.15.194:0/2621320632 shutdown_connections 2024-04-26T09:41:27.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.748+0000 7fde48597700 1 -- 172.21.15.194:0/2621320632 wait complete. 2024-04-26T09:41:27.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde48597700 1 Processor -- start 2024-04-26T09:41:27.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde48597700 1 -- start start 2024-04-26T09:41:27.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde48597700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065430 0x7fde40133010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:27.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde48597700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde40065f20 0x7fde40133500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:27.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde48597700 1 --2- >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde401271f0 0x7fde40086f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-04-26T09:41:27.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde48597700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde4008a600 con 0x7fde40065f20 2024-04-26T09:41:27.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde48597700 1 -- --> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde40089c80 con 0x7fde40065430 2024-04-26T09:41:27.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde48597700 1 -- --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_getmap magic: 0 v1 -- 0x7fde4008a830 con 0x7fde401271f0 2024-04-26T09:41:27.757 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde45b32700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde40065f20 0x7fde40133500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:27.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde46333700 1 --2- >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065430 0x7fde40133010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:27.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde45b32700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde40065f20 0x7fde40133500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:48894/0 (socket says 172.21.15.194:48894) 2024-04-26T09:41:27.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde45b32700 1 -- 172.21.15.194:0/3498600851 learned_addr learned my addr 172.21.15.194:0/3498600851 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-26T09:41:27.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.749+0000 7fde46b34700 1 --2- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde401271f0 0x7fde40086f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-04-26T09:41:27.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde46b34700 1 -- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065430 msgr2=0x7fde40133010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde46b34700 1 --2- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065430 0x7fde40133010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde46b34700 1 -- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde40065f20 msgr2=0x7fde40133500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde46b34700 1 --2- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde40065f20 0x7fde40133500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde46b34700 1 -- 172.21.15.194:0/3498600851 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c0092a0 con 0x7fde401271f0 2024-04-26T09:41:27.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde45b32700 1 --2- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde40065f20 0x7fde40133500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:27.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde46333700 1 --2- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065430 0x7fde40133010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2024-04-26T09:41:27.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde46b34700 1 --2- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde401271f0 0x7fde40086f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fde38014000 tx=0x7fde38009780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T09:41:27.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde36ffd700 1 -- 172.21.15.194:0/3498600851 <== mon.2 v2:172.21.15.194:3302/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde38012410 con 0x7fde401271f0 2024-04-26T09:41:27.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde36ffd700 1 -- 172.21.15.194:0/3498600851 <== mon.2 v2:172.21.15.194:3302/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3801d630 con 0x7fde401271f0 2024-04-26T09:41:27.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde48597700 1 -- 172.21.15.194:0/3498600851 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde38009a20 con 0x7fde401271f0 2024-04-26T09:41:27.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde36ffd700 1 -- 172.21.15.194:0/3498600851 <== mon.2 v2:172.21.15.194:3302/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde380070e0 con 0x7fde401271f0 2024-04-26T09:41:27.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.750+0000 7fde48597700 1 -- 172.21.15.194:0/3498600851 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde401339f0 con 0x7fde401271f0 2024-04-26T09:41:27.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.751+0000 7fde48597700 1 -- 172.21.15.194:0/3498600851 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde40093600 con 0x7fde401271f0 2024-04-26T09:41:27.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.751+0000 7fde36ffd700 1 -- 172.21.15.194:0/3498600851 <== mon.2 v2:172.21.15.194:3302/0 4 ==== mgrmap(e 4) v1 ==== 48491+0+0 (secure 0 0 0) 0x7fde38003040 con 0x7fde401271f0 2024-04-26T09:41:27.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.751+0000 7fde36ffd700 1 -- 172.21.15.194:0/3498600851 <== mon.2 v2:172.21.15.194:3302/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2838+0+0 (secure 0 0 0) 0x7fde38056dc0 con 0x7fde401271f0 2024-04-26T09:41:27.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.755+0000 7fde36ffd700 1 -- 172.21.15.194:0/3498600851 <== mon.2 v2:172.21.15.194:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+131111 (secure 0 0 0) 0x7fde3803af80 con 0x7fde401271f0 2024-04-26T09:41:27.896 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.894+0000 7fde48597700 1 -- 172.21.15.194:0/3498600851 --> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fde40069090 con 0x7fde401271f0 2024-04-26T09:41:27.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.898+0000 7fde36ffd700 1 -- 172.21.15.194:0/3498600851 <== mon.2 v2:172.21.15.194:3302/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fde3803af80 con 0x7fde401271f0 2024-04-26T09:41:27.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.899+0000 7fde36ffd700 1 -- 172.21.15.194:0/3498600851 <== mon.2 v2:172.21.15.194:3302/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7fde3803b160 con 0x7fde401271f0 2024-04-26T09:41:27.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.901+0000 7fde48597700 1 -- 172.21.15.194:0/3498600851 >> v1:172.21.15.194:6816/72244 conn(0x7fde24041bb0 legacy=0x7fde24044070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.901+0000 7fde48597700 1 -- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde401271f0 msgr2=0x7fde40086f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T09:41:27.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.901+0000 7fde48597700 1 --2- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde401271f0 0x7fde40086f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fde38014000 tx=0x7fde38009780).stop 2024-04-26T09:41:27.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.901+0000 7fde48597700 1 -- 172.21.15.194:0/3498600851 shutdown_connections 2024-04-26T09:41:27.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.901+0000 7fde48597700 1 --2- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3302/0,v1:172.21.15.194:6791/0] conn(0x7fde401271f0 0x7fde40086f40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.901+0000 7fde48597700 1 --2- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3301/0,v1:172.21.15.194:6790/0] conn(0x7fde40065430 0x7fde40133010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.901+0000 7fde48597700 1 --2- 172.21.15.194:0/3498600851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde40065f20 0x7fde40133500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-04-26T09:41:27.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.901+0000 7fde48597700 1 -- 172.21.15.194:0/3498600851 >> 172.21.15.194:0/3498600851 conn(0x7fde40094ce0 msgr2=0x7fde40067890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T09:41:27.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.901+0000 7fde48597700 1 -- 172.21.15.194:0/3498600851 shutdown_connections 2024-04-26T09:41:27.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-26T09:41:27.901+0000 7fde48597700 1 -- 172.21.15.194:0/3498600851 wait complete. 2024-04-26T09:41:27.915 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-26T09:41:27.915 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-26T09:41:27.915 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-26T09:41:27.915 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T09:41:27.946 INFO:tasks.ceph.osd.0:Stopped 2024-04-26T09:41:27.947 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-26T09:41:27.947 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T09:41:28.008 INFO:tasks.ceph.osd.1:Stopped 2024-04-26T09:41:28.008 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-26T09:41:28.008 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T09:41:28.072 INFO:tasks.ceph.osd.2:Stopped 2024-04-26T09:41:28.073 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-26T09:41:28.073 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T09:41:28.138 INFO:tasks.ceph.osd.3:Stopped 2024-04-26T09:41:28.138 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-26T09:41:28.138 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-26T09:41:28.138 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T09:41:28.196 INFO:tasks.ceph.mgr.x:Stopped 2024-04-26T09:41:28.197 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-26T09:41:28.197 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T09:41:28.254 INFO:tasks.ceph.mgr.y:Stopped 2024-04-26T09:41:28.254 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-26T09:41:28.254 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T09:41:28.255 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T09:41:28.304 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T09:41:28.305 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T09:41:28.305 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T09:41:28.354 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T09:41:28.355 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T09:41:28.355 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T09:41:28.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-26T09:41:28.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-26T09:41:28.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-26T09:41:28.404 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-26T09:41:28.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-04-26T09:41:28.405 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T09:41:28.405 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-26T09:41:28.405 DEBUG:teuthology.orchestra.run.smithi194:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'overall HEALTH_' | egrep -v '\(FS_' | egrep -v '\(MDS_' | egrep -v '\(OSD_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(CACHE_POOL_' | egrep -v '\(POOL_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'Monitor daemon marked osd' | egrep -v 'Behind on trimming' | egrep -v 'Manager daemon' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'scrub mismatch' | egrep -v ScrubResult | egrep -v 'wrongly marked' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_MSGR2_NOT_ENABLED\)' | egrep -v 'slow request' | head -n 1 2024-04-26T09:41:28.481 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:41:28.482 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-26T09:41:28.864 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:41:28.865 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-26T09:41:28.962 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:41:28.963 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-26T09:41:29.035 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:41:29.035 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-26T09:41:29.085 INFO:tasks.ceph:Archiving mon data... 2024-04-26T09:41:29.086 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639/data/mon.a.tgz 2024-04-26T09:41:29.087 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-04-26T09:41:29.104 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.dHyEiJ5qdn 2024-04-26T09:41:29.105 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.dHyEiJ5qdn 2024-04-26T09:41:29.266 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.dHyEiJ5qdn 2024-04-26T09:41:29.354 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.dHyEiJ5qdn is 297KB 2024-04-26T09:41:29.448 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.dHyEiJ5qdn 2024-04-26T09:41:29.466 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639/data/mon.b.tgz 2024-04-26T09:41:29.466 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-04-26T09:41:29.522 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.j5Zk6oGWEA 2024-04-26T09:41:29.523 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.j5Zk6oGWEA 2024-04-26T09:41:29.748 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.j5Zk6oGWEA 2024-04-26T09:41:29.835 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.j5Zk6oGWEA is 778KB 2024-04-26T09:41:29.940 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.j5Zk6oGWEA 2024-04-26T09:41:29.958 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639/data/mon.c.tgz 2024-04-26T09:41:29.958 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-04-26T09:41:30.015 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.09Dlab3cdP 2024-04-26T09:41:30.015 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.09Dlab3cdP 2024-04-26T09:41:30.242 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.09Dlab3cdP 2024-04-26T09:41:30.329 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.09Dlab3cdP is 778KB 2024-04-26T09:41:30.431 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.09Dlab3cdP 2024-04-26T09:41:30.449 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-26T09:41:30.449 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T09:41:30.514 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-04-26T09:41:30.515 DEBUG:teuthology.orchestra.run.smithi194:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-04-26T09:41:30.581 INFO:teuthology.orchestra.run.smithi194.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-04-26T09:41:30.587 INFO:tasks.ceph:Archiving crash dumps... 2024-04-26T09:41:30.589 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639/remote/smithi194/crash 2024-04-26T09:41:30.590 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T09:41:30.654 INFO:tasks.ceph:Compressing logs... 2024-04-26T09:41:30.654 DEBUG:teuthology.orchestra.run.smithi194:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T09:41:31.302 INFO:tasks.ceph:Archiving logs... 2024-04-26T09:41:31.303 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639/remote/smithi194/log 2024-04-26T09:41:31.306 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T09:41:31.455 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T09:41:31.469 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T09:41:31.470 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T09:41:31.525 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T09:41:31.535 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T09:41:31.535 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T09:41:31.553 INFO:teuthology.orchestra.run.smithi194.stderr:bash: ntpq: command not found 2024-04-26T09:41:31.558 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-26T09:41:31.558 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-04-26T09:41:31.558 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv01.front.sepia.ceph.com 3 6 377 52 +1365us[+1379us] +/- 47ms 2024-04-26T09:41:31.558 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 52 -1824us[-1824us] +/- 79ms 2024-04-26T09:41:31.558 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 -374us[ -374us] +/- 86ms 2024-04-26T09:41:31.559 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-26T09:41:31.559 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T09:41:31.568 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T09:41:31.568 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T09:41:31.578 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-26T09:41:31.662 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-26T09:41:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T09:41:31.691 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T09:41:31.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T09:41:31.783 INFO:teuthology.task.internal:Duration was 1574.574956 seconds 2024-04-26T09:41:31.783 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T09:41:31.793 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T09:41:31.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T09:41:31.839 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-26T09:41:32.102 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T09:41:32.102 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-04-26T09:41:32.103 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T09:41:32.125 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T09:41:32.125 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T09:41:32.208 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T09:41:32.208 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T09:41:32.316 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T09:41:32.327 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T09:41:32.327 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T09:41:32.354 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T09:41:32.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T09:41:32.421 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-04-26T09:41:32.444 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T09:41:32.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T09:41:32.501 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T09:41:32.512 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T09:41:32.513 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639/remote/smithi194 2024-04-26T09:41:32.513 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T09:41:32.579 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T09:41:32.579 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T09:41:32.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T09:41:32.642 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T09:41:32.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T09:41:32.658 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T09:41:32.658 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T09:41:32.688 INFO:teuthology.orchestra.run.smithi194.stdout: 525364 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 09:41 /home/ubuntu/cephtest 2024-04-26T09:41:32.689 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T09:41:32.704 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T09:41:32.733 INFO:teuthology.nuke:Checking targets against current locks 2024-04-26T09:41:32.759 DEBUG:teuthology.nuke:shortname: smithi194 2024-04-26T09:41:32.759 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T09:41:32.783 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_22:24:15-fs-reef-distro-default-smithi/7672639', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 08:16:59.501868', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAWlc2kZ/DO1NfCuDq1WvvsV03cE++vlE+/MHCAreg73mQ6TuDVK3Ctt9MmSJJpn8zoPeSeXJ6LxMHohlkrJac8='} 2024-04-26T09:41:32.839 INFO:teuthology.orchestra.console.smithi194:Power off 2024-04-26T09:41:32.839 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-26T09:41:32.861 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-04-26T09:41:32.862 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T09:41:32.893 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-04-26T09:41:33.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-26T09:41:33.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-26T09:41:33.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-26T09:41:33.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-26T09:41:33.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-04-26T09:41:36.894 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T09:41:37.003 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-04-26T09:41:41.007 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T09:41:41.119 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-04-26T09:41:45.121 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-26T09:41:45.233 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-04-26T09:41:45.334 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-04-26T09:41:45.438 INFO:teuthology.run:Summary data: description: fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} duration: 1574.5749561786652 failure_reason: ceph version 18.2.2-1254.g19b26365 was not installed, found 16.2.15-0.el8. flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fa69cedd5e4f4ceb9fce36a9f085c19c status: fail success: false 2024-04-26T09:41:45.438 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T09:41:45.524 INFO:teuthology.run:FAIL