2024-09-21T03:17:03.690 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T03:17:03.711 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T03:17:03.795 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913465 branch: wip-pdonnell-testing-20240920.212106-debug description: fs/upgrade/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/squid 1-upgrade}} email: pdonnell@redhat.com first_in_suite: false job_id: '7913465' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'install ceph/squid latest ' name: pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240920.212106-debug ceph: cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: client mount timeout: 30 debug client: 20 debug mgr: 20 debug ms: 1 mon warn on pool no app: false mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: global: bluestore warn on legacy statfs: false bluestore warn on no per pool omap: false mon pg warn min per osd: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon warn on osd down out interval zero: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - scrub mismatch - ScrubResult - wrongly marked - \(POOL_APP_NOT_ENABLED\) - \(SLOW_OPS\) - overall HEALTH_ - \(MON_MSGR2_NOT_ENABLED\) - slow request sha1: 1843221895adccfb3b692060251dd6a0183bc410 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1843221895adccfb3b692060251dd6a0183bc410 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240920.212106-debug sha1: 1843221895adccfb3b692060251dd6a0183bc410 owner: scheduled_pdonnell@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 seed: 7864 sha1: 1843221895adccfb3b692060251dd6a0183bc410 sleep_before_teardown: 0 subset: 67/512 suite: fs suite_branch: wip-pdonnell-testing-20240920.212106-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1843221895adccfb3b692060251dd6a0183bc410 targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIYnl57E9wIeGhKEI21JeJ1LN17/9krtRzRjly6d8kFvEEOyy9MRNrkGqFAL2cMji+4/jIw1L0/JUZdTCYjvv6Q= tasks: - install: branch: squid exclude_packages: - librados3 - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - cephadm - ceph-volume extra_packages: - librados2 - print: '**** done installing squid' - ceph: conf: global: mon warn on pool no app: false ms bind msgr2: false log-ignorelist: - overall HEALTH_ - \(FS_ - \(MDS_ - \(OSD_ - \(MON_DOWN\) - \(CACHE_POOL_ - \(POOL_ - \(MGR_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - Monitor daemon marked osd - Behind on trimming - Manager daemon - exec: osd.0: - ceph osd set-require-min-compat-client squid - print: '**** done ceph' - print: '*** upgrading, no cephfs present' - exec: mon.a: - ceph fs dump - install.upgrade: mon.a: null - print: '**** done install.upgrade' - ceph.restart: daemons: - mon.* - mgr.* mon-health-to-clog: false wait-for-healthy: false - ceph.healthy: null - ceph.restart: daemons: - osd.* wait-for-healthy: false wait-for-osds-up: true - exec: mon.a: - ceph versions - ceph osd dump -f json-pretty - ceph fs dump - ceph osd require-osd-release squid - for f in `ceph osd pool ls` ; do ceph osd pool set $f pg_autoscale_mode off ; done - ceph.healthy: null - print: '**** done ceph.restart' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_23:48:22 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T03:17:03.795 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa; will attempt to use it 2024-09-21T03:17:03.796 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa/tasks 2024-09-21T03:17:03.796 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T03:17:03.797 INFO:teuthology.task.internal:Checking packages... 2024-09-21T03:17:03.817 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1843221895adccfb3b692060251dd6a0183bc410' 2024-09-21T03:17:03.817 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T03:17:03.817 INFO:teuthology.packaging:ref: None 2024-09-21T03:17:03.817 INFO:teuthology.packaging:tag: None 2024-09-21T03:17:03.817 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240920.212106-debug 2024-09-21T03:17:03.817 INFO:teuthology.packaging:sha1: 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T03:17:03.817 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240920.212106-debug 2024-09-21T03:17:03.956 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5120.g18432218 2024-09-21T03:17:03.957 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T03:17:03.965 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T03:17:03.965 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T03:17:04.002 INFO:teuthology.task.internal:Saving configuration 2024-09-21T03:17:04.019 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T03:17:04.028 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T03:17:04.052 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913465', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 03:10:58.356741', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIYnl57E9wIeGhKEI21JeJ1LN17/9krtRzRjly6d8kFvEEOyy9MRNrkGqFAL2cMji+4/jIw1L0/JUZdTCYjvv6Q='} 2024-09-21T03:17:04.052 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T03:17:04.059 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-09-21T03:17:04.059 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T03:17:04.092 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efc1bd24160>, signals=[15]) 2024-09-21T03:17:04.092 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T03:17:04.100 INFO:teuthology.task.internal:Opening connections... 2024-09-21T03:17:04.100 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:17:04.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T03:17:04.179 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T03:17:04.186 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2024-09-21T03:17:04.273 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2024-09-21T03:17:04.273 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2024-09-21T03:17:04.331 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream" 2024-09-21T03:17:04.331 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="9" 2024-09-21T03:17:04.331 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos" 2024-09-21T03:17:04.331 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora" 2024-09-21T03:17:04.331 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="9" 2024-09-21T03:17:04.332 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el9" 2024-09-21T03:17:04.332 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T03:17:04.332 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2024-09-21T03:17:04.332 INFO:teuthology.orchestra.run.smithi027.stdout:LOGO="fedora-logo-icon" 2024-09-21T03:17:04.332 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T03:17:04.332 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/" 2024-09-21T03:17:04.332 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T03:17:04.332 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T03:17:04.332 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T03:17:04.333 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2024-09-21T03:17:04.361 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T03:17:04.370 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T03:17:04.378 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T03:17:04.378 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2024-09-21T03:17:04.397 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T03:17:04.404 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T03:17:04.404 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2024-09-21T03:17:04.462 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T03:17:04.498 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T03:17:04.512 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T03:17:04.512 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T03:17:04.514 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T03:17:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T03:17:04.532 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-09-21T03:17:04.591 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-503.el9.x86_64 2024-09-21T03:17:04.592 INFO:teuthology.task.kernel:Running kernel on smithi027: 5.14.0-503.el9.x86_64 2024-09-21T03:17:04.592 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2024-09-21T03:17:08.062 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.3 MB 00:01 2024-09-21T03:17:10.321 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 18 MB/s | 20 MB 00:01 2024-09-21T03:17:15.972 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-21T03:17:17.581 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 19 kB 00:00 2024-09-21T03:17:18.710 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:01 2024-09-21T03:17:24.324 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-21T03:17:26.415 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T03:17:26.415 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T03:17:26.504 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:17:26.519 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:17:26.519 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-09-21T03:17:26.519 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:17:26.519 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-21T03:17:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T03:17:26.520 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-21T03:17:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T03:17:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T03:17:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T03:17:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:17:26.520 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:17:26.520 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:17:26.520 INFO:teuthology.orchestra.run.smithi027.stdout:Install 4 Packages 2024-09-21T03:17:26.520 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:17:26.521 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 79 M 2024-09-21T03:17:26.521 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 126 M 2024-09-21T03:17:26.521 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:17:27.101 INFO:teuthology.orchestra.run.smithi027.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 65 kB/s | 36 kB 00:00 2024-09-21T03:17:28.676 INFO:teuthology.orchestra.run.smithi027.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-21T03:17:28.918 INFO:teuthology.orchestra.run.smithi027.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 6.6 MB/s | 16 MB 00:02 2024-09-21T03:17:31.185 INFO:teuthology.orchestra.run.smithi027.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 7.0 MB/s | 29 MB 00:04 2024-09-21T03:17:31.185 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:17:31.186 INFO:teuthology.orchestra.run.smithi027.stdout:Total 17 MB/s | 79 MB 00:04 2024-09-21T03:17:31.900 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:17:32.312 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:17:32.313 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:17:33.324 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:17:33.325 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:17:35.907 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:17:36.598 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T03:17:36.679 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T03:17:39.560 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T03:17:39.698 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T03:17:44.919 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T03:17:44.965 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T03:17:50.194 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T03:18:30.345 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T03:18:30.387 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T03:18:30.698 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T03:18:30.698 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T03:18:30.698 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T03:18:30.698 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T03:18:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T03:18:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:18:31.542 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:18:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T03:18:31.542 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T03:18:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T03:18:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T03:18:31.543 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:18:31.543 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:18:31.955 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T03:18:32.034 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T03:18:32.035 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T03:18:32.035 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T03:18:32.036 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T03:18:32.036 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T03:18:32.036 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T03:18:32.036 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T03:18:32.036 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T03:18:32.096 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T03:18:32.097 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T03:18:32.097 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2024-09-21T03:18:32.139 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2024-09-21T03:18:32.139 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2024-09-21T03:18:32.139 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.06 2024-09-21T03:18:32.139 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 82.el9 2024-09-21T03:18:32.139 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2024-09-21T03:18:32.139 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T03:18:32.139 INFO:teuthology.orchestra.run.smithi027.stdout:Group : Unspecified 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 8274670 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : builder@centos.org 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : CentOS 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T03:18:32.140 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T03:18:32.141 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2024-09-21T03:18:32.141 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:18:32.141 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T03:18:32.141 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T03:18:32.141 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 5.14.0-508.el9.x86_64 2024-09-21T03:18:32.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T03:18:32.787 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2024-09-21T03:18:34.109 INFO:teuthology.orchestra.run.smithi027.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T03:18:34.137 INFO:teuthology.orchestra.run.smithi027.stderr:done 2024-09-21T03:18:34.139 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-09-21T03:18:34.155 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.0GkBzRaSYU 2024-09-21T03:18:34.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0GkBzRaSYU 2024-09-21T03:18:34.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.0GkBzRaSYU 2024-09-21T03:18:34.361 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.0GkBzRaSYU is 6KB 2024-09-21T03:18:34.381 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.0GkBzRaSYU 2024-09-21T03:18:34.396 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T03:18:34.459 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T03:18:34.459 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T03:18:34.459 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T03:18:34.459 INFO:teuthology.orchestra.run.smithi027.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T03:18:34.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T03:18:34.616 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2024-09-21T03:19:04.646 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T03:19:04.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-21T03:19:04.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T03:19:23.119 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-21T03:19:32.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-21T03:19:32.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T03:19:35.187 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-21T03:19:47.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-21T03:19:47.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T03:19:50.255 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-21T03:20:05.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-21T03:20:05.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T03:20:08.335 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-21T03:20:26.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-21T03:20:26.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T03:20:44.783 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2024-09-21T03:21:05.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-21T03:21:05.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T03:21:06.190 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-09-21T03:21:06.567 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-21T03:21:06.568 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T03:21:06.568 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2024-09-21T03:21:06.584 INFO:teuthology.orchestra.run.smithi027.stdout:5.14.0-508.el9.x86_64 2024-09-21T03:21:06.584 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T03:21:06.584 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T03:21:06.585 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T03:21:07.586 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T03:21:07.586 DEBUG:teuthology.orchestra.run.smithi027:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T03:21:07.663 INFO:teuthology.orchestra.run.smithi027.stdout:ttyS1 2024-09-21T03:21:07.688 DEBUG:teuthology.parallel:result is None 2024-09-21T03:21:07.688 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T03:21:07.695 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T03:21:07.695 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T03:21:07.714 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T03:21:07.721 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T03:21:07.732 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T03:21:07.732 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T03:21:07.793 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T03:21:07.799 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T03:21:07.799 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T03:21:07.866 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T03:21:07.878 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T03:21:07.879 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T03:21:07.886 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T03:21:07.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T03:21:07.949 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T03:21:07.957 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T03:21:07.958 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T03:21:08.007 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T03:21:08.089 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T03:21:08.139 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T03:21:08.221 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T03:21:08.265 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:21:08.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T03:21:08.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2024-09-21T03:21:08.418 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T03:21:08.749 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T03:21:08.757 INFO:teuthology.task.internal:Starting timer... 2024-09-21T03:21:08.757 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T03:21:08.767 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T03:21:08.774 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-21T03:21:08.775 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2024-09-21T03:21:08.821 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: 2024-09-21T03:21:08.823 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T03:21:08.823 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2024-09-21T03:21:08.856 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2024-09-21T03:21:08.856 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive'} 2024-09-21T03:21:08.857 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T03:21:08.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T03:21:08.924 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T03:21:08.924 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2024-09-21T03:21:09.004 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T03:21:09.014 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-21T03:21:09.016 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-21T03:21:09.122 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T03:21:09.132 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T03:21:09.132 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T03:24:55.849 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com')] 2024-09-21T03:24:55.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-21T03:24:55.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T03:24:55.928 DEBUG:teuthology.orchestra.run.smithi027:> true 2024-09-21T03:24:56.012 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2024-09-21T03:24:56.012 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T03:24:56.023 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T03:24:56.023 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T03:24:56.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T03:24:56.085 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T03:24:56.101 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T03:24:56.129 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2024-09-21T03:24:56.141 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2024-09-21T03:24:56.157 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T03:24:56.172 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T03:24:56.238 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-09-21T03:24:56.242 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T03:24:56.242 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-09-21T03:24:56.242 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T03:24:56.242 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T03:24:56.242 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T03:24:56.242 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T03:24:56.243 INFO:teuthology.run_tasks:Running task install... 2024-09-21T03:24:56.251 DEBUG:teuthology.task.install:project ceph 2024-09-21T03:24:56.251 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'}} 2024-09-21T03:24:56.251 DEBUG:teuthology.task.install:config {'branch': 'squid', 'exclude_packages': ['librados3', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'cephadm', 'ceph-volume'], 'extra_packages': ['librados2'], 'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'} 2024-09-21T03:24:56.251 INFO:teuthology.task.install:Using flavor: default 2024-09-21T03:24:56.256 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T03:24:56.256 INFO:teuthology.task.install:extra packages: ['librados2'] 2024-09-21T03:24:56.256 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['librados3', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'cephadm', 'ceph-volume'], 'extra_packages': ['librados2'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T03:24:56.257 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T03:24:56.257 INFO:teuthology.packaging:ref: None 2024-09-21T03:24:56.257 INFO:teuthology.packaging:tag: None 2024-09-21T03:24:56.257 INFO:teuthology.packaging:branch: squid 2024-09-21T03:24:56.257 INFO:teuthology.packaging:sha1: 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T03:24:56.257 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-21T03:24:56.469 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/ 2024-09-21T03:24:56.469 INFO:teuthology.task.install.rpm:Package version is 19.1.1-373.g4e45e6b4 2024-09-21T03:24:56.581 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T03:24:56.582 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:24:56.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T03:24:56.609 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, librados2 on remote rpm x86_64 2024-09-21T03:24:56.609 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T03:24:56.609 INFO:teuthology.packaging:ref: None 2024-09-21T03:24:56.609 INFO:teuthology.packaging:tag: None 2024-09-21T03:24:56.609 INFO:teuthology.packaging:branch: squid 2024-09-21T03:24:56.609 INFO:teuthology.packaging:sha1: 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T03:24:56.609 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T03:24:56.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T03:24:56.752 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T03:24:56.818 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-09-21T03:24:56.819 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-09-21T03:24:57.061 INFO:teuthology.orchestra.run.smithi027.stdout:45 files removed 2024-09-21T03:24:57.087 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw 2024-09-21T03:24:57.673 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 292 kB/s | 84 kB 00:00 2024-09-21T03:24:57.948 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 51 kB/s | 12 kB 00:00 2024-09-21T03:24:58.241 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 7.4 kB/s | 1.9 kB 00:00 2024-09-21T03:24:58.883 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-21T03:25:00.675 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-21T03:25:05.360 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.5 MB 00:00 2024-09-21T03:25:06.852 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 19 kB 00:00 2024-09-21T03:25:07.981 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:01 2024-09-21T03:25:13.579 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-21T03:25:15.926 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:25:15.943 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:25:15.943 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-21T03:25:15.943 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:25:15.943 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 11 M 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.4 M 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.2 M 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.4 M 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 21 M 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 25 k 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T03:25:15.944 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 751 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 507 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.3 M 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 45 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 136 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 162 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 321 k 2024-09-21T03:25:15.945 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 300 k 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 99 k 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout:Installing weak dependencies: 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout:Install 30 Packages 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 2 Packages 2024-09-21T03:25:15.946 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:15.947 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 60 M 2024-09-21T03:25:15.947 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:25:16.514 INFO:teuthology.orchestra.run.smithi027.stdout:(1/32): ceph-base-19.1.1-373.g4e45e6b4.el9.x86_ 12 MB/s | 5.4 MB 00:00 2024-09-21T03:25:16.572 INFO:teuthology.orchestra.run.smithi027.stdout:(2/32): ceph-selinux-19.1.1-373.g4e45e6b4.el9.x 433 kB/s | 25 kB 00:00 2024-09-21T03:25:16.664 INFO:teuthology.orchestra.run.smithi027.stdout:(3/32): libcephfs2-19.1.1-373.g4e45e6b4.el9.x86 8.0 MB/s | 751 kB 00:00 2024-09-21T03:25:16.748 INFO:teuthology.orchestra.run.smithi027.stdout:(4/32): libradosstriper1-19.1.1-373.g4e45e6b4.e 5.9 MB/s | 507 kB 00:00 2024-09-21T03:25:16.915 INFO:teuthology.orchestra.run.smithi027.stdout:(5/32): ceph-radosgw-19.1.1-373.g4e45e6b4.el9.x 12 MB/s | 11 MB 00:00 2024-09-21T03:25:16.965 INFO:teuthology.orchestra.run.smithi027.stdout:(6/32): python3-ceph-argparse-19.1.1-373.g4e45e 908 kB/s | 45 kB 00:00 2024-09-21T03:25:17.057 INFO:teuthology.orchestra.run.smithi027.stdout:(7/32): librgw2-19.1.1-373.g4e45e6b4.el9.x86_64 17 MB/s | 5.3 MB 00:00 2024-09-21T03:25:17.082 INFO:teuthology.orchestra.run.smithi027.stdout:(8/32): python3-ceph-common-19.1.1-373.g4e45e6b 1.1 MB/s | 136 kB 00:00 2024-09-21T03:25:17.108 INFO:teuthology.orchestra.run.smithi027.stdout:(9/32): python3-cephfs-19.1.1-373.g4e45e6b4.el9 3.2 MB/s | 162 kB 00:00 2024-09-21T03:25:17.133 INFO:teuthology.orchestra.run.smithi027.stdout:(10/32): python3-rados-19.1.1-373.g4e45e6b4.el9 6.2 MB/s | 321 kB 00:00 2024-09-21T03:25:17.158 INFO:teuthology.orchestra.run.smithi027.stdout:(11/32): python3-rbd-19.1.1-373.g4e45e6b4.el9.x 5.8 MB/s | 300 kB 00:00 2024-09-21T03:25:17.184 INFO:teuthology.orchestra.run.smithi027.stdout:(12/32): python3-rgw-19.1.1-373.g4e45e6b4.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-21T03:25:17.409 INFO:teuthology.orchestra.run.smithi027.stdout:(13/32): ceph-common-19.1.1-373.g4e45e6b4.el9.x 16 MB/s | 21 MB 00:01 2024-09-21T03:25:17.976 INFO:teuthology.orchestra.run.smithi027.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 80 kB/s | 45 kB 00:00 2024-09-21T03:25:18.009 INFO:teuthology.orchestra.run.smithi027.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 126 kB/s | 107 kB 00:00 2024-09-21T03:25:18.043 INFO:teuthology.orchestra.run.smithi027.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 186 kB/s | 160 kB 00:00 2024-09-21T03:25:18.627 INFO:teuthology.orchestra.run.smithi027.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 72 kB/s | 42 kB 00:00 2024-09-21T03:25:18.777 INFO:teuthology.orchestra.run.smithi027.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 381 kB/s | 292 kB 00:00 2024-09-21T03:25:18.844 INFO:teuthology.orchestra.run.smithi027.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 763 kB/s | 662 kB 00:00 2024-09-21T03:25:19.328 INFO:teuthology.orchestra.run.smithi027.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 65 kB/s | 36 kB 00:00 2024-09-21T03:25:19.378 INFO:teuthology.orchestra.run.smithi027.stdout:(21/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.1 MB/s | 308 kB 00:00 2024-09-21T03:25:19.403 INFO:teuthology.orchestra.run.smithi027.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 285 kB/s | 221 kB 00:00 2024-09-21T03:25:19.420 INFO:teuthology.orchestra.run.smithi027.stdout:(23/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 82 kB/s | 47 kB 00:00 2024-09-21T03:25:19.453 INFO:teuthology.orchestra.run.smithi027.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 528 kB/s | 26 kB 00:00 2024-09-21T03:25:19.487 INFO:teuthology.orchestra.run.smithi027.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 2.1 MB/s | 67 kB 00:00 2024-09-21T03:25:19.504 INFO:teuthology.orchestra.run.smithi027.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 646 kB/s | 49 kB 00:00 2024-09-21T03:25:19.554 INFO:teuthology.orchestra.run.smithi027.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-21T03:25:19.580 INFO:teuthology.orchestra.run.smithi027.stdout:(28/32): re2-20211101-20.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-21T03:25:19.647 INFO:teuthology.orchestra.run.smithi027.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-21T03:25:19.722 INFO:teuthology.orchestra.run.smithi027.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-21T03:25:19.922 INFO:teuthology.orchestra.run.smithi027.stdout:(31/32): librados2-19.1.1-373.g4e45e6b4.el9.x86 10 MB/s | 3.4 MB 00:00 2024-09-21T03:25:19.973 INFO:teuthology.orchestra.run.smithi027.stdout:(32/32): librbd1-19.1.1-373.g4e45e6b4.el9.x86_6 9.7 MB/s | 3.2 MB 00:00 2024-09-21T03:25:19.979 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:25:19.979 INFO:teuthology.orchestra.run.smithi027.stdout:Total 15 MB/s | 60 MB 00:04 2024-09-21T03:25:20.089 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:25:20.130 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:25:20.131 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:25:20.848 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:25:20.848 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:25:22.382 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:25:22.460 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-21T03:25:22.495 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-21T03:25:22.520 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-21T03:25:22.542 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-21T03:25:22.569 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-21T03:25:22.770 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-21T03:25:22.832 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-21T03:25:23.236 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-21T03:25:23.288 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-21T03:25:23.385 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-21T03:25:23.480 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 8/34 2024-09-21T03:25:23.515 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 9/34 2024-09-21T03:25:23.578 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 9/34 2024-09-21T03:25:23.615 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-21T03:25:23.659 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 11/34 2024-09-21T03:25:23.729 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-21T03:25:23.758 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-21T03:25:23.823 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-21T03:25:23.857 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-21T03:25:23.969 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-21T03:25:24.000 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-21T03:25:24.050 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-21T03:25:24.082 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-21T03:25:24.111 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-21T03:25:24.144 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-21T03:25:24.479 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-21T03:25:24.523 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-21T03:25:24.598 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-21T03:25:24.628 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/34 2024-09-21T03:25:24.664 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-21T03:25:25.174 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-21T03:25:25.201 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-21T03:25:25.273 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-21T03:25:25.353 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/34 2024-09-21T03:25:25.378 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 28/34 2024-09-21T03:25:27.863 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-21T03:25:27.900 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-21T03:25:28.514 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-21T03:25:28.562 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-21T03:25:28.645 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-21T03:25:28.646 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T03:25:28.646 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T03:25:28.646 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:28.680 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-21T03:25:39.108 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-21T03:25:39.109 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2024-09-21T03:25:39.109 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2024-09-21T03:25:39.109 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2024-09-21T03:25:39.109 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2024-09-21T03:25:39.109 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2024-09-21T03:25:39.109 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /root 2024-09-21T03:25:39.109 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2024-09-21T03:25:39.109 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:40.130 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-21T03:25:40.158 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-21T03:25:40.158 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T03:25:40.158 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T03:25:40.158 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T03:25:40.158 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T03:25:40.158 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:40.160 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-21T03:25:40.249 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-21T03:25:40.249 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T03:25:41.405 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 8/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 9/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 10/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-21T03:25:41.406 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-21T03:25:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-21T03:25:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-21T03:25:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-21T03:25:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-21T03:25:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-21T03:25:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-21T03:25:41.409 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-21T03:25:41.410 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-21T03:25:41.410 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-21T03:25:41.410 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-21T03:25:41.410 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-21T03:25:41.410 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-21T03:25:41.410 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-21T03:25:41.410 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-21T03:25:41.410 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-21T03:25:41.411 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-21T03:25:41.411 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-21T03:25:41.411 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-21T03:25:41.411 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/34 2024-09-21T03:25:42.333 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T03:25:42.333 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:42.333 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-09-21T03:25:42.333 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.333 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.333 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:25:42.333 INFO:teuthology.orchestra.run.smithi027.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T03:25:42.334 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T03:25:42.335 INFO:teuthology.orchestra.run.smithi027.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T03:25:42.336 INFO:teuthology.orchestra.run.smithi027.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T03:25:42.336 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T03:25:42.336 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:42.336 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:25:42.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-test 2024-09-21T03:25:43.294 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:30 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:25:43.432 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:25:43.432 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:25:43.432 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 49 M 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 163 k 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout:Install 4 Packages 2024-09-21T03:25:43.433 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:43.434 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 49 M 2024-09-21T03:25:43.434 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 209 M 2024-09-21T03:25:43.434 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:25:43.682 INFO:teuthology.orchestra.run.smithi027.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-21T03:25:43.707 INFO:teuthology.orchestra.run.smithi027.stdout:(2/4): libcephsqlite-19.1.1-373.g4e45e6b4.el9.x 734 kB/s | 163 kB 00:00 2024-09-21T03:25:43.732 INFO:teuthology.orchestra.run.smithi027.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T03:25:45.049 INFO:teuthology.orchestra.run.smithi027.stdout:(4/4): ceph-test-19.1.1-373.g4e45e6b4.el9.x86_6 31 MB/s | 49 MB 00:01 2024-09-21T03:25:45.051 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:25:45.051 INFO:teuthology.orchestra.run.smithi027.stdout:Total 30 MB/s | 49 MB 00:01 2024-09-21T03:25:45.079 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:25:45.101 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:25:45.102 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:25:45.491 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:25:45.491 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:25:46.041 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:25:46.119 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T03:25:46.165 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T03:25:46.193 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/4 2024-09-21T03:25:51.055 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/4 2024-09-21T03:25:51.145 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/4 2024-09-21T03:25:53.549 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/4 2024-09-21T03:25:53.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/4 2024-09-21T03:25:53.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/4 2024-09-21T03:25:53.549 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T03:25:53.865 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T03:25:53.865 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:53.866 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:25:53.866 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:53.866 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:25:53.866 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T03:25:53.866 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T03:25:53.866 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:53.866 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:25:54.177 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph 2024-09-21T03:25:54.788 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:41 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:25:54.934 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:25:54.937 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:25:54.937 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-21T03:25:54.937 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:25:54.937 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-21T03:25:54.937 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 6.6 k 2024-09-21T03:25:54.937 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-21T03:25:54.937 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 2.4 M 2024-09-21T03:25:54.937 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 1.5 M 2024-09-21T03:25:54.937 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 249 k 2024-09-21T03:25:54.937 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 4.7 M 2024-09-21T03:25:54.937 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 17 M 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T03:25:54.938 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T03:25:54.939 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T03:25:54.940 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T03:25:54.940 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T03:25:54.940 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T03:25:54.940 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T03:25:54.940 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:54.940 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:25:54.940 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:25:54.940 INFO:teuthology.orchestra.run.smithi027.stdout:Install 37 Packages 2024-09-21T03:25:54.940 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:25:54.941 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 30 M 2024-09-21T03:25:54.941 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 108 M 2024-09-21T03:25:54.941 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:25:55.788 INFO:teuthology.orchestra.run.smithi027.stdout:(1/37): ceph-19.1.1-373.g4e45e6b4.el9.x86_64.rp 46 kB/s | 6.6 kB 00:00 2024-09-21T03:25:55.955 INFO:teuthology.orchestra.run.smithi027.stdout:(2/37): ceph-mgr-19.1.1-373.g4e45e6b4.el9.x86_6 4.9 MB/s | 1.5 MB 00:00 2024-09-21T03:25:56.005 INFO:teuthology.orchestra.run.smithi027.stdout:(3/37): ceph-mds-19.1.1-373.g4e45e6b4.el9.x86_6 6.6 MB/s | 2.4 MB 00:00 2024-09-21T03:25:56.089 INFO:teuthology.orchestra.run.smithi027.stdout:(4/37): ceph-mgr-modules-core-19.1.1-373.g4e45e 2.9 MB/s | 249 kB 00:00 2024-09-21T03:25:56.181 INFO:teuthology.orchestra.run.smithi027.stdout:(5/37): ceph-mon-19.1.1-373.g4e45e6b4.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-09-21T03:25:56.248 INFO:teuthology.orchestra.run.smithi027.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-09-21T03:25:56.314 INFO:teuthology.orchestra.run.smithi027.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-09-21T03:25:56.365 INFO:teuthology.orchestra.run.smithi027.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 6.7 MB/s | 1.2 MB 00:00 2024-09-21T03:25:56.390 INFO:teuthology.orchestra.run.smithi027.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.8 MB/s | 135 kB 00:00 2024-09-21T03:25:56.565 INFO:teuthology.orchestra.run.smithi027.stdout:(10/37): ceph-osd-19.1.1-373.g4e45e6b4.el9.x86_ 28 MB/s | 17 MB 00:00 2024-09-21T03:25:56.590 INFO:teuthology.orchestra.run.smithi027.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 553 kB/s | 125 kB 00:00 2024-09-21T03:25:56.624 INFO:teuthology.orchestra.run.smithi027.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 917 kB/s | 215 kB 00:00 2024-09-21T03:25:56.666 INFO:teuthology.orchestra.run.smithi027.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-21T03:25:56.683 INFO:teuthology.orchestra.run.smithi027.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 594 kB/s | 35 kB 00:00 2024-09-21T03:25:56.725 INFO:teuthology.orchestra.run.smithi027.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-21T03:25:56.742 INFO:teuthology.orchestra.run.smithi027.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 869 kB/s | 51 kB 00:00 2024-09-21T03:25:56.767 INFO:teuthology.orchestra.run.smithi027.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 765 kB/s | 77 kB 00:00 2024-09-21T03:25:56.792 INFO:teuthology.orchestra.run.smithi027.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 621 kB/s | 42 kB 00:00 2024-09-21T03:25:56.835 INFO:teuthology.orchestra.run.smithi027.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-21T03:25:56.851 INFO:teuthology.orchestra.run.smithi027.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-09-21T03:25:56.868 INFO:teuthology.orchestra.run.smithi027.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 576 kB/s | 43 kB 00:00 2024-09-21T03:25:56.893 INFO:teuthology.orchestra.run.smithi027.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 606 kB/s | 11 kB 00:00 2024-09-21T03:25:56.919 INFO:teuthology.orchestra.run.smithi027.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-21T03:25:56.936 INFO:teuthology.orchestra.run.smithi027.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 443 kB/s | 18 kB 00:00 2024-09-21T03:25:56.961 INFO:teuthology.orchestra.run.smithi027.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 785 kB/s | 19 kB 00:00 2024-09-21T03:25:56.987 INFO:teuthology.orchestra.run.smithi027.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-21T03:25:57.011 INFO:teuthology.orchestra.run.smithi027.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-21T03:25:57.045 INFO:teuthology.orchestra.run.smithi027.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 789 kB/s | 46 kB 00:00 2024-09-21T03:25:57.071 INFO:teuthology.orchestra.run.smithi027.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-09-21T03:25:57.096 INFO:teuthology.orchestra.run.smithi027.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 662 kB/s | 16 kB 00:00 2024-09-21T03:25:57.129 INFO:teuthology.orchestra.run.smithi027.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-21T03:25:57.155 INFO:teuthology.orchestra.run.smithi027.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-21T03:25:57.180 INFO:teuthology.orchestra.run.smithi027.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-21T03:25:57.205 INFO:teuthology.orchestra.run.smithi027.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-21T03:25:57.231 INFO:teuthology.orchestra.run.smithi027.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-21T03:25:57.256 INFO:teuthology.orchestra.run.smithi027.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-21T03:25:57.281 INFO:teuthology.orchestra.run.smithi027.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-21T03:25:57.286 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:25:57.286 INFO:teuthology.orchestra.run.smithi027.stdout:Total 13 MB/s | 30 MB 00:02 2024-09-21T03:25:57.445 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:25:57.490 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:25:57.490 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:25:57.979 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:25:57.980 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:25:59.496 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:25:59.559 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T03:25:59.593 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T03:25:59.620 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T03:25:59.695 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T03:25:59.861 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T03:25:59.894 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T03:25:59.990 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T03:26:00.067 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T03:26:00.114 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T03:26:00.148 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T03:26:00.178 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T03:26:00.221 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T03:26:00.255 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T03:26:00.305 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T03:26:00.356 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T03:26:00.390 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T03:26:00.440 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T03:26:00.479 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T03:26:00.556 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T03:26:00.707 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T03:26:00.791 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T03:26:00.849 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T03:26:00.896 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T03:26:00.954 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T03:26:01.026 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T03:26:01.224 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T03:26:01.283 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T03:26:01.365 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T03:26:01.462 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T03:26:01.507 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T03:26:01.628 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T03:26:01.802 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 32/37 2024-09-21T03:26:01.833 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/37 2024-09-21T03:26:01.861 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/37 2024-09-21T03:26:01.861 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T03:26:01.861 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T03:26:01.861 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T03:26:01.861 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T03:26:01.861 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:03.384 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 34/37 2024-09-21T03:26:03.413 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 34/37 2024-09-21T03:26:03.413 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T03:26:03.413 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T03:26:03.413 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T03:26:03.413 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T03:26:03.413 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:03.876 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/37 2024-09-21T03:26:03.902 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/37 2024-09-21T03:26:03.902 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T03:26:03.902 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T03:26:03.902 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T03:26:03.902 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T03:26:03.902 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:04.135 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/37 2024-09-21T03:26:04.160 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/37 2024-09-21T03:26:04.161 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T03:26:04.161 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T03:26:04.161 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T03:26:04.161 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T03:26:04.161 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:04.227 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/37 2024-09-21T03:26:06.068 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 6/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T03:26:06.069 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T03:26:06.070 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T03:26:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T03:26:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T03:26:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T03:26:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T03:26:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T03:26:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T03:26:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T03:26:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T03:26:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T03:26:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T03:26:06.072 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T03:26:07.126 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T03:26:07.127 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T03:26:07.128 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:07.129 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:07.480 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-base 2024-09-21T03:26:08.097 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:08.189 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T03:26:08.234 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:08.235 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-21T03:26:08.235 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:08.306 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T03:26:08.915 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:09.054 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:09.055 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:09.055 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-21T03:26:09.055 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:09.055 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-21T03:26:09.055 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 145 k 2024-09-21T03:26:09.055 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:09.055 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:26:09.055 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:09.055 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-21T03:26:09.055 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:09.056 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 145 k 2024-09-21T03:26:09.056 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 435 k 2024-09-21T03:26:09.056 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:26:09.245 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-immutable-object-cache-19.1.1-373.g4e45e6b 767 kB/s | 145 kB 00:00 2024-09-21T03:26:09.246 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:26:09.246 INFO:teuthology.orchestra.run.smithi027.stdout:Total 761 kB/s | 145 kB 00:00 2024-09-21T03:26:09.246 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:26:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:26:09.254 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:26:09.316 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:26:09.316 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:26:09.469 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:26:09.663 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-21T03:26:09.690 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-21T03:26:09.690 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T03:26:09.691 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T03:26:09.691 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:10.667 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-21T03:26:10.668 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:10.668 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:26:10.668 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:26:10.668 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:10.668 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:10.868 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-mgr 2024-09-21T03:26:11.473 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:58 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:11.565 INFO:teuthology.orchestra.run.smithi027.stdout:Package ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T03:26:11.611 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:11.612 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-21T03:26:11.612 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:11.677 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-fuse 2024-09-21T03:26:12.281 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:12.418 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:12.419 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:12.419 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-21T03:26:12.419 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:12.419 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-21T03:26:12.419 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 897 k 2024-09-21T03:26:12.419 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-21T03:26:12.419 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T03:26:12.419 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:12.419 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:26:12.419 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:12.419 INFO:teuthology.orchestra.run.smithi027.stdout:Install 2 Packages 2024-09-21T03:26:12.420 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:12.420 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 976 k 2024-09-21T03:26:12.420 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 2.9 M 2024-09-21T03:26:12.420 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:26:12.708 INFO:teuthology.orchestra.run.smithi027.stdout:(1/2): ceph-fuse-19.1.1-373.g4e45e6b4.el9.x86_6 3.3 MB/s | 897 kB 00:00 2024-09-21T03:26:13.075 INFO:teuthology.orchestra.run.smithi027.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 126 kB/s | 80 kB 00:00 2024-09-21T03:26:13.075 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:26:13.076 INFO:teuthology.orchestra.run.smithi027.stdout:Total 1.5 MB/s | 976 kB 00:00 2024-09-21T03:26:13.088 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:26:13.112 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:26:13.112 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:26:13.201 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:26:13.201 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:26:13.395 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:26:13.556 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T03:26:13.633 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-21T03:26:14.323 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-21T03:26:14.323 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/2 2024-09-21T03:26:14.601 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T03:26:14.602 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:14.602 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:26:14.602 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T03:26:14.602 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:14.602 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:14.863 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados-devel 2024-09-21T03:26:15.476 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:02 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:15.614 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:15.615 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:15.615 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-21T03:26:15.615 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:15.615 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-21T03:26:15.615 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 127 k 2024-09-21T03:26:15.615 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:15.615 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:26:15.615 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:15.616 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-21T03:26:15.616 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:15.616 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 127 k 2024-09-21T03:26:15.616 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 456 k 2024-09-21T03:26:15.616 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:26:15.796 INFO:teuthology.orchestra.run.smithi027.stdout:librados-devel-19.1.1-373.g4e45e6b4.el9.x86_64. 705 kB/s | 127 kB 00:00 2024-09-21T03:26:15.868 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:26:15.869 INFO:teuthology.orchestra.run.smithi027.stdout:Total 698 kB/s | 127 kB 00:00 2024-09-21T03:26:15.869 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:26:15.869 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:26:15.869 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:26:15.869 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:26:15.869 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:26:16.016 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:26:16.136 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:16.539 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:16.786 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:26:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:26:16.786 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:16.786 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:17.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs2 2024-09-21T03:26:17.619 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:04 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:17.714 INFO:teuthology.orchestra.run.smithi027.stdout:Package libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T03:26:17.758 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:17.760 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-21T03:26:17.760 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:17.825 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install libcephfs-devel 2024-09-21T03:26:18.428 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:05 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:18.565 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:18.566 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:18.566 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-21T03:26:18.567 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:18.567 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-21T03:26:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 32 k 2024-09-21T03:26:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:18.567 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:26:18.567 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:18.567 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-21T03:26:18.567 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:18.567 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 32 k 2024-09-21T03:26:18.567 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 142 k 2024-09-21T03:26:18.567 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:26:18.725 INFO:teuthology.orchestra.run.smithi027.stdout:libcephfs-devel-19.1.1-373.g4e45e6b4.el9.x86_64 201 kB/s | 32 kB 00:00 2024-09-21T03:26:18.725 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:26:18.726 INFO:teuthology.orchestra.run.smithi027.stdout:Total 200 kB/s | 32 kB 00:00 2024-09-21T03:26:18.726 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:26:18.729 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:26:18.729 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:26:18.747 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:26:18.747 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:26:18.832 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:26:18.948 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:19.297 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:19.523 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:19.523 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:19.524 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:26:19.524 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:26:19.524 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:19.524 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:19.708 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados2 2024-09-21T03:26:20.312 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:07 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:20.404 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T03:26:20.449 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:20.450 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-21T03:26:20.450 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:20.517 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librbd1 2024-09-21T03:26:21.114 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:08 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:21.206 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T03:26:21.250 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:21.251 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-21T03:26:21.252 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:21.328 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rados 2024-09-21T03:26:21.931 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:08 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:22.023 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T03:26:22.067 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:22.069 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-21T03:26:22.069 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:22.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rgw 2024-09-21T03:26:22.737 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:09 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:22.830 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T03:26:22.875 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:22.876 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-21T03:26:22.876 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:22.942 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-cephfs 2024-09-21T03:26:23.540 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:10 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:23.633 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T03:26:23.677 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:23.678 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-21T03:26:23.678 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:23.750 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install python3-rbd 2024-09-21T03:26:24.362 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:11 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:24.456 INFO:teuthology.orchestra.run.smithi027.stdout:Package python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T03:26:24.501 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:24.502 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-21T03:26:24.502 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:24.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-fuse 2024-09-21T03:26:25.190 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:12 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:25.327 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:25.327 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:25.327 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-09-21T03:26:25.328 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:25.328 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-21T03:26:25.328 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 85 k 2024-09-21T03:26:25.328 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:25.328 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:26:25.328 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:25.328 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-21T03:26:25.328 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:25.329 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 85 k 2024-09-21T03:26:25.329 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 231 k 2024-09-21T03:26:25.329 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:26:25.520 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-fuse-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 447 kB/s | 85 kB 00:00 2024-09-21T03:26:25.521 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:26:25.521 INFO:teuthology.orchestra.run.smithi027.stdout:Total 442 kB/s | 85 kB 00:00 2024-09-21T03:26:25.522 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:26:25.529 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:26:25.529 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:26:25.589 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:26:25.590 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:26:25.733 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:26:25.844 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:26.227 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:26.465 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:26:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:26:26.465 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:26.465 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:26.716 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-mirror 2024-09-21T03:26:27.318 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:14 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:27.456 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:27.457 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:27.457 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repo Size 2024-09-21T03:26:27.458 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:27.458 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-21T03:26:27.458 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-21T03:26:27.458 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:27.458 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:26:27.458 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:27.459 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-21T03:26:27.459 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:27.459 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 3.1 M 2024-09-21T03:26:27.459 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 13 M 2024-09-21T03:26:27.459 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:26:27.783 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-mirror-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 9.6 MB/s | 3.1 MB 00:00 2024-09-21T03:26:27.784 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:26:27.784 INFO:teuthology.orchestra.run.smithi027.stdout:Total 9.5 MB/s | 3.1 MB 00:00 2024-09-21T03:26:27.784 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:26:27.794 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:26:27.795 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:26:27.873 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:26:27.874 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:26:28.417 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:26:28.476 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:28.503 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:28.503 INFO:teuthology.orchestra.run.smithi027.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T03:26:28.503 INFO:teuthology.orchestra.run.smithi027.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T03:26:28.503 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T03:26:28.504 INFO:teuthology.orchestra.run.smithi027.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T03:26:28.504 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:29.591 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:29.591 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:29.591 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:26:29.591 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:26:29.591 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:29.591 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:29.829 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install rbd-nbd 2024-09-21T03:26:30.443 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:17 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:30.580 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:30.580 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:30.580 INFO:teuthology.orchestra.run.smithi027.stdout: Package Architecture Version Repository Size 2024-09-21T03:26:30.581 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:30.581 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2024-09-21T03:26:30.581 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 170 k 2024-09-21T03:26:30.581 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:30.581 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:26:30.581 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:26:30.581 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-21T03:26:30.581 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:30.581 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 170 k 2024-09-21T03:26:30.581 INFO:teuthology.orchestra.run.smithi027.stdout:Installed size: 494 k 2024-09-21T03:26:30.581 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:26:30.791 INFO:teuthology.orchestra.run.smithi027.stdout:rbd-nbd-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 829 kB/s | 170 kB 00:00 2024-09-21T03:26:30.791 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:26:30.791 INFO:teuthology.orchestra.run.smithi027.stdout:Total 823 kB/s | 170 kB 00:00 2024-09-21T03:26:30.791 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:26:30.796 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:26:30.796 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:26:30.856 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:26:30.856 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:26:31.009 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:26:31.144 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:31.568 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:31.815 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-21T03:26:31.815 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:31.815 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:26:31.815 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-21T03:26:31.815 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:31.815 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:32.057 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install librados2 2024-09-21T03:26:32.665 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:19 ago on Sat 21 Sep 2024 03:25:13 AM UTC. 2024-09-21T03:26:32.758 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-21T03:26:32.803 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:26:32.803 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2024-09-21T03:26:32.803 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:26:32.876 DEBUG:teuthology.parallel:result is None 2024-09-21T03:26:32.876 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T03:26:32.876 INFO:teuthology.packaging:ref: None 2024-09-21T03:26:32.876 INFO:teuthology.packaging:tag: None 2024-09-21T03:26:32.876 INFO:teuthology.packaging:branch: squid 2024-09-21T03:26:32.876 INFO:teuthology.packaging:sha1: 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T03:26:32.876 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-21T03:26:33.119 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T03:26:33.141 INFO:teuthology.orchestra.run.smithi027.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-21T03:26:33.141 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373.g4e45e6b4.el9 2024-09-21T03:26:33.141 INFO:teuthology.task.install:The correct ceph version 19.1.1-373.g4e45e6b4 is installed. 2024-09-21T03:26:33.143 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T03:26:33.143 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:33.143 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T03:26:33.207 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T03:26:33.207 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:33.207 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T03:26:33.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T03:26:33.338 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T03:26:33.338 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:33.339 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T03:26:33.404 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T03:26:33.470 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T03:26:33.471 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:33.471 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T03:26:33.536 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T03:26:33.601 INFO:teuthology.run_tasks:Running task print... 2024-09-21T03:26:33.611 INFO:teuthology.task.print:**** done installing squid 2024-09-21T03:26:33.611 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T03:26:33.686 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-21T03:26:33.686 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2024-09-21T03:26:33.712 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-21T03:26:33.712 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T03:26:33.777 INFO:tasks.ceph:Creating extra log directories... 2024-09-21T03:26:33.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T03:26:33.851 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-21T03:26:33.851 INFO:tasks.ceph:config {'conf': {'global': {'mon warn on pool no app': False, 'ms bind msgr2': False, 'bluestore warn on legacy statfs': False, 'bluestore warn on no per pool omap': False, 'mon pg warn min per osd': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon warn on osd down out interval zero': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['overall HEALTH_', '\\(FS_', '\\(MDS_', '\\(OSD_', '\\(MON_DOWN\\)', '\\(CACHE_POOL_', '\\(POOL_', '\\(MGR_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'Monitor daemon marked osd', 'Behind on trimming', 'Manager daemon', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'scrub mismatch', 'ScrubResult', 'wrongly marked', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'overall HEALTH_', '\\(MON_MSGR2_NOT_ENABLED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-21T03:26:33.851 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913465', 'branch': 'wip-pdonnell-testing-20240920.212106-debug', 'description': 'fs/upgrade/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/squid 1-upgrade}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7913465', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'install ceph/squid latest\n'}], 'name': 'pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240920.212106-debug'}, 'ceph': {'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'global': {'bluestore warn on legacy statfs': False, 'bluestore warn on no per pool omap': False, 'mon pg warn min per osd': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon warn on osd down out interval zero': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'scrub mismatch', 'ScrubResult', 'wrongly marked', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'overall HEALTH_', '\\(MON_MSGR2_NOT_ENABLED\\)', 'slow request'], 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240920.212106-debug', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 90, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 7864, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'sleep_before_teardown': 0, 'subset': '67/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240920.212106-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'targets': {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIYnl57E9wIeGhKEI21JeJ1LN17/9krtRzRjly6d8kFvEEOyy9MRNrkGqFAL2cMji+4/jIw1L0/JUZdTCYjvv6Q='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'branch': 'squid', 'exclude_packages': ['librados3', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'cephadm', 'ceph-volume'], 'extra_packages': ['librados2'], 'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'}}, {'print': '**** done installing squid'}, {'ceph': {'conf': {'global': {'mon warn on pool no app': False, 'ms bind msgr2': False, 'bluestore warn on legacy statfs': False, 'bluestore warn on no per pool omap': False, 'mon pg warn min per osd': 0}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon warn on osd down out interval zero': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'log-ignorelist': ['overall HEALTH_', '\\(FS_', '\\(MDS_', '\\(OSD_', '\\(MON_DOWN\\)', '\\(CACHE_POOL_', '\\(POOL_', '\\(MGR_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'Monitor daemon marked osd', 'Behind on trimming', 'Manager daemon', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'scrub mismatch', 'ScrubResult', 'wrongly marked', '\\(POOL_APP_NOT_ENABLED\\)', '\\(SLOW_OPS\\)', 'overall HEALTH_', '\\(MON_MSGR2_NOT_ENABLED\\)', 'slow request'], 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'cluster': 'ceph'}}, {'exec': {'osd.0': ['ceph osd set-require-min-compat-client squid']}}, {'print': '**** done ceph'}, {'print': '*** upgrading, no cephfs present'}, {'exec': {'mon.a': ['ceph fs dump']}}, {'install.upgrade': {'mon.a': None}}, {'print': '**** done install.upgrade'}, {'ceph.restart': {'daemons': ['mon.*', 'mgr.*'], 'mon-health-to-clog': False, 'wait-for-healthy': False}}, {'ceph.healthy': None}, {'ceph.restart': {'daemons': ['osd.*'], 'wait-for-healthy': False, 'wait-for-osds-up': True}}, {'exec': {'mon.a': ['ceph versions', 'ceph osd dump -f json-pretty', 'ceph fs dump', 'ceph osd require-osd-release squid', 'for f in `ceph osd pool ls` ; do ceph osd pool set $f pg_autoscale_mode off ; done']}}, {'ceph.healthy': None}, {'print': '**** done ceph.restart'}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-20_23:48:22', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T03:26:33.852 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T03:26:33.899 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T03:26:33.965 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:33.965 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T03:26:34.021 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T03:26:34.022 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2024-09-21T03:26:34.077 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T03:26:34.077 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T03:26:34.078 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-21T03:26:34.078 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T03:26:34.078 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T03:26:34.078 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-21 03:26:28.750692873 +0000 2024-09-21T03:26:34.078 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-21 03:24:36.776784716 +0000 2024-09-21T03:26:34.078 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-21 03:24:36.776784716 +0000 2024-09-21T03:26:34.078 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-21 03:24:36.776784716 +0000 2024-09-21T03:26:34.078 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T03:26:34.142 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-21T03:26:34.142 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-21T03:26:34.142 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000213608 s, 2.4 MB/s 2024-09-21T03:26:34.143 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T03:26:34.200 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2024-09-21T03:26:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T03:26:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T03:26:34.257 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-21T03:26:34.257 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T03:26:34.257 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T03:26:34.257 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-21 03:26:28.750692873 +0000 2024-09-21T03:26:34.257 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-21 03:24:37.113775978 +0000 2024-09-21T03:26:34.257 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-21 03:24:37.113775978 +0000 2024-09-21T03:26:34.257 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-21 03:24:37.113775978 +0000 2024-09-21T03:26:34.258 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T03:26:34.321 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-21T03:26:34.321 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-21T03:26:34.321 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000210381 s, 2.4 MB/s 2024-09-21T03:26:34.322 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T03:26:34.379 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2024-09-21T03:26:34.435 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T03:26:34.435 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T03:26:34.435 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-21T03:26:34.435 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T03:26:34.435 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T03:26:34.435 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-21 03:26:28.751692847 +0000 2024-09-21T03:26:34.435 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-21 03:24:37.445767369 +0000 2024-09-21T03:26:34.435 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-21 03:24:37.445767369 +0000 2024-09-21T03:26:34.435 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-21 03:24:37.445767369 +0000 2024-09-21T03:26:34.436 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T03:26:34.498 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-21T03:26:34.498 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-21T03:26:34.499 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.00020644 s, 2.5 MB/s 2024-09-21T03:26:34.500 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T03:26:34.556 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2024-09-21T03:26:34.613 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T03:26:34.613 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T03:26:34.613 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-21T03:26:34.613 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T03:26:34.613 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T03:26:34.613 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2024-09-21 03:26:28.751692847 +0000 2024-09-21T03:26:34.613 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2024-09-21 03:24:37.786758528 +0000 2024-09-21T03:26:34.613 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2024-09-21 03:24:37.786758528 +0000 2024-09-21T03:26:34.613 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: 2024-09-21 03:24:37.786758528 +0000 2024-09-21T03:26:34.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T03:26:34.676 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2024-09-21T03:26:34.676 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2024-09-21T03:26:34.676 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000184289 s, 2.8 MB/s 2024-09-21T03:26:34.678 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T03:26:34.734 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T03:26:34.734 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-21T03:26:34.735 INFO:tasks.ceph:Generating config... 2024-09-21T03:26:34.736 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-21T03:26:34.736 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-21T03:26:34.736 INFO:tasks.ceph:[global] bluestore warn on legacy statfs = False 2024-09-21T03:26:34.736 INFO:tasks.ceph:[global] bluestore warn on no per pool omap = False 2024-09-21T03:26:34.736 INFO:tasks.ceph:[global] mon pg warn min per osd = 0 2024-09-21T03:26:34.736 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T03:26:34.736 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T03:26:34.736 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-21T03:26:34.736 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-21T03:26:34.736 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-21T03:26:34.736 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-09-21T03:26:34.736 INFO:tasks.ceph:[mon] mon warn on osd down out interval zero = False 2024-09-21T03:26:34.737 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-21T03:26:34.737 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-21T03:26:34.737 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-21T03:26:34.737 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-21T03:26:34.737 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-21T03:26:34.737 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-21T03:26:34.737 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-21T03:26:34.737 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-21T03:26:34.738 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-21T03:26:34.738 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-21T03:26:34.738 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-21T03:26:34.738 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-21T03:26:34.738 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-21T03:26:34.738 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-21T03:26:34.738 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-21T03:26:34.738 INFO:tasks.ceph:Setting up mon.a... 2024-09-21T03:26:34.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-21T03:26:34.828 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.keyring 2024-09-21T03:26:34.831 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-21T03:26:34.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T03:26:34.902 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.27'), ('mon.b', '[v2:172.21.15.27:3301,v1:172.21.15.27:6790]'), ('mon.c', '[v2:172.21.15.27:3302,v1:172.21.15.27:6791]')] 2024-09-21T03:26:34.903 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27,[v2:172.21.15.27:3301,v1:172.21.15.27:6790],[v2:172.21.15.27:3302,v1:172.21.15.27:6791]', 'mon warn on pool no app': False, 'ms bind msgr2': False, 'bluestore warn on legacy statfs': False, 'bluestore warn on no per pool omap': False, 'mon pg warn min per osd': 0}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon warn on osd down out interval zero': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-21T03:26:34.904 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:34.904 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T03:26:34.959 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --addv b '[v2:172.21.15.27:3301,v1:172.21.15.27:6790]' --addv c '[v2:172.21.15.27:3302,v1:172.21.15.27:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-21T03:26:35.046 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-21T03:26:35.046 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid 22a90953-a6e7-4790-ab76-aa25cec033b2 2024-09-21T03:26:35.046 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = quincy 2024-09-21T03:26:35.046 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2024-09-21T03:26:35.046 INFO:teuthology.orchestra.run.smithi027.stdout:fsid 22a90953-a6e7-4790-ab76-aa25cec033b2 2024-09-21T03:26:35.047 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2024-09-21T03:26:35.045764+0000 2024-09-21T03:26:35.047 INFO:teuthology.orchestra.run.smithi027.stdout:created 2024-09-21T03:26:35.045764+0000 2024-09-21T03:26:35.047 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 17 (quincy) 2024-09-21T03:26:35.047 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2024-09-21T03:26:35.047 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2024-09-21T03:26:35.047 INFO:teuthology.orchestra.run.smithi027.stdout:1: [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] mon.b 2024-09-21T03:26:35.047 INFO:teuthology.orchestra.run.smithi027.stdout:2: [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] mon.c 2024-09-21T03:26:35.047 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-21T03:26:35.048 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T03:26:35.063 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 22a90953-a6e7-4790-ab76-aa25cec033b2... 2024-09-21T03:26:35.064 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T03:26:35.147 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.148 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2024-09-21T03:26:35.149 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.150 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,[v2:172.21.15.27:3301,v1:172.21.15.27:6790],[v2:172.21.15.27:3302,v1:172.21.15.27:6791]" 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on pool no app = False 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: ms bind msgr2 = False 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore warn on legacy statfs = False 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore warn on no per pool omap = False 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn min per osd = 0 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 22a90953-a6e7-4790-ab76-aa25cec033b2 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2024-09-21T03:26:35.151 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug trim objects = true 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2024-09-21T03:26:35.152 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2024-09-21T03:26:35.153 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-09-21T03:26:35.154 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2024-09-21T03:26:35.155 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: mon down mkfs grace = 300 2024-09-21T03:26:35.156 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = False 2024-09-21T03:26:35.157 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:35.157 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2024-09-21T03:26:35.157 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2024-09-21T03:26:35.157 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2024-09-21T03:26:35.157 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2024-09-21T03:26:35.157 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T03:26:35.157 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T03:26:35.157 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2024-09-21T03:26:35.157 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2024-09-21T03:26:35.157 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2024-09-21T03:26:35.159 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-21T03:26:35.159 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-21T03:26:35.215 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T03:26:35.215 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:35.215 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-21T03:26:35.231 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:35.231 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-21T03:26:35.286 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:26:35.286 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:35.286 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T03:26:35.287 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T03:26:35.361 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:35.361 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T03:26:35.417 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T03:26:35.417 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T03:26:35.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T03:26:35.509 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T03:26:35.512 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-21T03:26:35.564 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-21T03:26:35.566 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T03:26:35.567 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T03:26:35.567 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T03:26:35.567 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-21T03:26:35.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-21T03:26:35.592 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T03:26:35.592 INFO:tasks.ceph:role: osd.0 2024-09-21T03:26:35.592 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:26:35.592 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T03:26:35.665 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T03:26:35.665 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T03:26:35.665 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T03:26:35.665 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T03:26:35.665 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T03:26:35.665 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-09-21T03:26:35.665 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T03:26:35.665 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T03:26:35.666 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T03:26:35.666 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T03:26:36.169 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-09-21T03:26:36.171 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-09-21T03:26:36.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-21T03:26:36.304 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-21T03:26:36.358 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-21T03:26:36.426 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T03:26:36.427 INFO:tasks.ceph:role: osd.1 2024-09-21T03:26:36.427 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:26:36.427 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T03:26:36.501 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T03:26:36.501 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T03:26:36.501 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T03:26:36.502 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T03:26:36.502 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T03:26:36.502 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-09-21T03:26:36.502 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T03:26:36.502 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T03:26:36.502 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T03:26:36.502 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T03:26:36.943 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-09-21T03:26:36.945 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-09-21T03:26:36.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-21T03:26:37.037 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-21T03:26:37.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-21T03:26:37.133 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T03:26:37.134 INFO:tasks.ceph:role: osd.2 2024-09-21T03:26:37.134 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:26:37.134 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T03:26:37.206 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T03:26:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T03:26:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T03:26:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T03:26:37.207 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T03:26:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-09-21T03:26:37.207 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T03:26:37.207 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T03:26:37.207 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T03:26:37.207 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T03:26:37.692 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-09-21T03:26:37.694 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-09-21T03:26:37.694 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-21T03:26:37.845 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-21T03:26:37.877 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-21T03:26:37.942 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T03:26:37.943 INFO:tasks.ceph:role: osd.3 2024-09-21T03:26:37.943 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:26:37.943 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-21T03:26:38.015 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T03:26:38.016 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T03:26:38.016 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T03:26:38.016 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T03:26:38.016 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T03:26:38.016 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2024-09-21T03:26:38.016 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T03:26:38.016 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T03:26:38.016 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T03:26:38.016 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T03:26:38.518 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2024-09-21T03:26:38.520 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2024-09-21T03:26:38.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-21T03:26:38.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-21T03:26:38.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T03:26:38.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:38.789+0000 7f455ea1d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-21T03:26:38.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:38.790+0000 7f455ea1d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-21T03:26:38.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:38.790+0000 7f455ea1d640 -1 bdev(0x55d990985800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-21T03:26:38.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:38.790+0000 7f455ea1d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-21T03:26:41.714 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T03:26:41.745 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T03:26:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:41.838+0000 7f2c35302640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-21T03:26:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:41.838+0000 7f2c35302640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-21T03:26:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:41.838+0000 7f2c35302640 -1 bdev(0x5560ddf1b800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-21T03:26:41.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:41.839+0000 7f2c35302640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-21T03:26:45.006 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T03:26:45.036 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T03:26:45.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:45.127+0000 7f9062e0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-21T03:26:45.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:45.128+0000 7f9062e0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-21T03:26:45.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:45.128+0000 7f9062e0d640 -1 bdev(0x5648f8b87800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-21T03:26:45.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:45.128+0000 7f9062e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-21T03:26:49.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T03:26:49.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T03:26:49.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:49.171+0000 7f245e60e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-21T03:26:49.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:49.171+0000 7f245e60e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-21T03:26:49.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:49.171+0000 7f245e60e640 -1 bdev(0x55c873d1d800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-21T03:26:49.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:49.171+0000 7f245e60e640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-21T03:26:52.332 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T03:26:52.364 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T03:26:52.365 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:52.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-21T03:26:52.435 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:52.435 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-21T03:26:52.504 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:52.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-21T03:26:52.573 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:52.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-21T03:26:52.641 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:52.641 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-21T03:26:52.708 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:52.708 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-21T03:26:52.774 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T03:26:52.774 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T03:26:52.839 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2024-09-21T03:26:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDrPO5mYtNYHhAAb0rh56TIMhj9/xscTzKeqQ== 2024-09-21T03:26:52.839 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.y] 2024-09-21T03:26:52.839 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDrPO5mxjmaIRAAI0AJa9m4cq0L0eR9y+QowA== 2024-09-21T03:26:52.840 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2024-09-21T03:26:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDuPO5maIkrLxAAl1B2+lkR0YWlViXxGlEZQA== 2024-09-21T03:26:52.840 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2024-09-21T03:26:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQDxPO5m7EgCMhAAv3olWHyMEeJqG3SL4t8Okw== 2024-09-21T03:26:52.840 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2024-09-21T03:26:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQD1PO5mcJ6mBxAAgznIp/KaX9Yy8Fhr3KBOYA== 2024-09-21T03:26:52.840 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.3] 2024-09-21T03:26:52.840 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQD5PO5m49E/ChAAd18PG9FcOd83f68n8GCjoQ== 2024-09-21T03:26:52.841 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T03:26:52.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T03:26:52.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T03:26:53.038 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T03:26:53.090 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T03:26:53.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T03:26:53.194 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T03:26:53.194 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-21T03:26:53.219 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T03:26:53.317 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-21T03:26:53.342 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-21T03:26:53.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T03:26:53.507 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-21T03:26:53.532 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-21T03:26:53.599 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T03:26:53.697 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-21T03:26:53.723 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T03:26:53.778 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-21T03:26:53.779 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-21T03:26:53.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-21T03:26:53.821 INFO:tasks.ceph.mon.a:Started 2024-09-21T03:26:53.821 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-21T03:26:53.821 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-21T03:26:53.823 INFO:tasks.ceph.mon.b:Started 2024-09-21T03:26:53.823 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-21T03:26:53.824 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-21T03:26:53.826 INFO:tasks.ceph.mon.c:Started 2024-09-21T03:26:53.826 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-21T03:26:53.826 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-21T03:26:53.826 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-21T03:26:53.828 INFO:tasks.ceph.mgr.x:Started 2024-09-21T03:26:53.828 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-21T03:26:53.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-21T03:26:53.830 INFO:tasks.ceph.mgr.y:Started 2024-09-21T03:26:53.830 INFO:tasks.ceph:setting config [client] client_mount_timeout = 600 2024-09-21T03:26:53.831 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set client client_mount_timeout 600 2024-09-21T03:26:53.833 INFO:tasks.ceph:setting config [client] debug_client = 20 2024-09-21T03:26:53.833 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set client debug_client 20 2024-09-21T03:26:53.877 INFO:tasks.ceph:setting config [client] debug_ms = 1 2024-09-21T03:26:53.878 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set client debug_ms 1 2024-09-21T03:26:53.923 INFO:tasks.ceph:setting config [client] rados_mon_op_timeout = 900 2024-09-21T03:26:53.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set client rados_mon_op_timeout 900 2024-09-21T03:26:53.966 INFO:tasks.ceph:setting config [client] rados_osd_op_timeout = 900 2024-09-21T03:26:53.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set client rados_osd_op_timeout 900 2024-09-21T03:26:54.008 INFO:tasks.ceph:setting config [mds] debug_mds = 20 2024-09-21T03:26:54.009 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mds debug_mds 20 2024-09-21T03:26:54.052 INFO:tasks.ceph:setting config [mds] debug_mds_balancer = 20 2024-09-21T03:26:54.052 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mds debug_mds_balancer 20 2024-09-21T03:26:54.096 INFO:tasks.ceph:setting config [mds] debug_ms = 2 2024-09-21T03:26:54.096 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mds debug_ms 2 2024-09-21T03:26:54.140 INFO:tasks.ceph:setting config [mds] mds_debug_frag = True 2024-09-21T03:26:54.140 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mds mds_debug_frag True 2024-09-21T03:26:54.183 INFO:tasks.ceph:setting config [mds] mds_debug_scatterstat = True 2024-09-21T03:26:54.183 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mds mds_debug_scatterstat True 2024-09-21T03:26:54.226 INFO:tasks.ceph:setting config [mds] mds_op_complaint_time = 180 2024-09-21T03:26:54.226 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mds mds_op_complaint_time 180 2024-09-21T03:26:54.269 INFO:tasks.ceph:setting config [mds] mds_verify_scatter = True 2024-09-21T03:26:54.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mds mds_verify_scatter True 2024-09-21T03:26:54.313 INFO:tasks.ceph:setting config [mds] osd_op_complaint_time = 180 2024-09-21T03:26:54.313 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mds osd_op_complaint_time 180 2024-09-21T03:26:54.357 INFO:tasks.ceph:setting config [mds] rados_mon_op_timeout = 900 2024-09-21T03:26:54.357 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mds rados_mon_op_timeout 900 2024-09-21T03:26:54.400 INFO:tasks.ceph:setting config [mds] rados_osd_op_timeout = 900 2024-09-21T03:26:54.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mds rados_osd_op_timeout 900 2024-09-21T03:26:54.444 INFO:tasks.ceph:setting config [mgr] client_mount_timeout = 30 2024-09-21T03:26:54.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mgr client_mount_timeout 30 2024-09-21T03:26:54.460 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:54.458+0000 7f3c11c12040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T03:26:54.461 INFO:tasks.ceph:setting config [mgr] debug_client = 20 2024-09-21T03:26:54.461 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mgr debug_client 20 2024-09-21T03:26:54.474 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:54.473+0000 7f2af1337040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T03:26:54.477 INFO:tasks.ceph:setting config [mgr] debug_mgr = 20 2024-09-21T03:26:54.477 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mgr debug_mgr 20 2024-09-21T03:26:54.496 INFO:tasks.ceph:setting config [mgr] debug_ms = 1 2024-09-21T03:26:54.496 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mgr debug_ms 1 2024-09-21T03:26:54.542 INFO:tasks.ceph:setting config [mgr] mon_warn_on_pool_no_app = False 2024-09-21T03:26:54.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mgr mon_warn_on_pool_no_app False 2024-09-21T03:26:54.568 INFO:tasks.ceph:setting config [mon] mon_op_complaint_time = 120 2024-09-21T03:26:54.568 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set mon mon_op_complaint_time 120 2024-09-21T03:26:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.599+0000 7f3bf9ffb640 1 -- 172.21.15.27:0/1818024896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c14091310 msgr2=0x7f3c14123290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.599+0000 7f3bf9ffb640 1 --2- 172.21.15.27:0/1818024896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c14091310 0x7f3c14123290 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400d9e0 tx=0x7f3c0400dea0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.603+0000 7f3bf9ffb640 1 -- 172.21.15.27:0/1818024896 shutdown_connections 2024-09-21T03:26:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.603+0000 7f3bf9ffb640 1 --2- 172.21.15.27:0/1818024896 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3c14096e40 0x7f3c14123d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.603+0000 7f3bf9ffb640 1 --2- 172.21.15.27:0/1818024896 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3c14095c10 0x7f3c141237d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.603+0000 7f3bf9ffb640 1 --2- 172.21.15.27:0/1818024896 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c14091310 0x7f3c14123290 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.603+0000 7f3bf9ffb640 1 -- 172.21.15.27:0/1818024896 >> 172.21.15.27:0/1818024896 conn(0x7f3c1409c7b0 msgr2=0x7f3c140997f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.607 INFO:tasks.ceph:setting config [osd] osd_op_complaint_time = 180 2024-09-21T03:26:54.607 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set osd osd_op_complaint_time 180 2024-09-21T03:26:54.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.603+0000 7f3bf9ffb640 1 -- 172.21.15.27:0/1818024896 shutdown_connections 2024-09-21T03:26:54.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.603+0000 7f3bf9ffb640 1 -- 172.21.15.27:0/1818024896 wait complete. 2024-09-21T03:26:54.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.622+0000 7feaf8eb1640 1 -- 172.21.15.27:0/1663389796 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7feadc005910 msgr2=0x7feadc005d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.623 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.622+0000 7feaf8eb1640 1 --2- 172.21.15.27:0/1663389796 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7feadc005910 0x7feadc005d90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feaf406e3d0 tx=0x7feadc006940 comp rx=0 tx=0).stop 2024-09-21T03:26:54.626 DEBUG:tasks.ceph:set 22 configs 2024-09-21T03:26:54.626 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi027') 2024-09-21T03:26:54.626 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi027') 2024-09-21T03:26:54.626 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi027') 2024-09-21T03:26:54.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.623+0000 7feaf8eb1640 1 -- 172.21.15.27:0/1663389796 shutdown_connections 2024-09-21T03:26:54.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.623+0000 7feaf8eb1640 1 --2- 172.21.15.27:0/1663389796 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7feadc006500 0x7feadc00cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.623+0000 7feaf8eb1640 1 --2- 172.21.15.27:0/1663389796 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7feadc005910 0x7feadc005d90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.623+0000 7feaf8eb1640 1 --2- 172.21.15.27:0/1663389796 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feadc004af0 0x7feadc004f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.623+0000 7feaf8eb1640 1 -- 172.21.15.27:0/1663389796 >> 172.21.15.27:0/1663389796 conn(0x7feaf409c930 msgr2=0x7feaf409cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.629+0000 7feaf8eb1640 1 -- 172.21.15.27:0/1663389796 shutdown_connections 2024-09-21T03:26:54.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.634+0000 7feaf8eb1640 1 -- 172.21.15.27:0/1663389796 wait complete. 2024-09-21T03:26:54.643 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:54.642+0000 7f3c11c12040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T03:26:54.648 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi027') 2024-09-21T03:26:54.649 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi027') 2024-09-21T03:26:54.649 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi027') 2024-09-21T03:26:54.649 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi027') 2024-09-21T03:26:54.649 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi027') 2024-09-21T03:26:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.651+0000 7feaf8eb1640 1 Processor -- start 2024-09-21T03:26:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.654+0000 7feaf8eb1640 1 -- start start 2024-09-21T03:26:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.654+0000 7feaf8eb1640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7feadc004af0 0x7feaf41250b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.654+0000 7feaf8eb1640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7feadc006500 0x7feaf41255f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.654+0000 7feaf8eb1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaf412a560 0x7feaf4125b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.654+0000 7feaf8eb1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7feaf412a3e0 con 0x7feaf412a560 2024-09-21T03:26:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.654+0000 7feaf8eb1640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7feaf412a260 con 0x7feadc004af0 2024-09-21T03:26:54.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.654+0000 7feaf8eb1640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7feaf412a9c0 con 0x7feadc006500 2024-09-21T03:26:54.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.655+0000 7feaf2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaf412a560 0x7feaf4125b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.655+0000 7feaf2d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaf412a560 0x7feaf4125b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42940/0 (socket says 172.21.15.27:42940) 2024-09-21T03:26:54.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.655+0000 7feaf2d9d640 1 -- 172.21.15.27:0/2644196513 learned_addr learned my addr 172.21.15.27:0/2644196513 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:54.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.655+0000 7feaf259c640 1 --2- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7feadc004af0 0x7feaf41250b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.655+0000 7feaf259c640 1 -- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7feadc006500 msgr2=0x7feaf41255f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.655+0000 7feaf259c640 1 --2- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7feadc006500 0x7feaf41255f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.655+0000 7feaf259c640 1 -- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaf412a560 msgr2=0x7feaf4125b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.655+0000 7feaf259c640 1 --2- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaf412a560 0x7feaf4125b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.655+0000 7feaf259c640 1 -- 172.21.15.27:0/2644196513 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf408b030 con 0x7feadc004af0 2024-09-21T03:26:54.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.655+0000 7feaf2d9d640 1 --2- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaf412a560 0x7feaf4125b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:26:54.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.656+0000 7feaf259c640 1 --2- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7feadc004af0 0x7feaf41250b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feaf406df70 tx=0x7feaec009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:54.660 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:54.657+0000 7f2af1337040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T03:26:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.657+0000 7feae2ffd640 1 -- 172.21.15.27:0/2644196513 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec019070 con 0x7feadc004af0 2024-09-21T03:26:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.657+0000 7feae2ffd640 1 -- 172.21.15.27:0/2644196513 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7feaec014440 con 0x7feadc004af0 2024-09-21T03:26:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.657+0000 7feae2ffd640 1 -- 172.21.15.27:0/2644196513 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaec014890 con 0x7feadc004af0 2024-09-21T03:26:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.658+0000 7feaf8eb1640 1 -- 172.21.15.27:0/2644196513 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaf4096a90 con 0x7feadc004af0 2024-09-21T03:26:54.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.658+0000 7feaf8eb1640 1 -- 172.21.15.27:0/2644196513 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feaf408c0f0 con 0x7feadc004af0 2024-09-21T03:26:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.660+0000 7feae2ffd640 1 -- 172.21.15.27:0/2644196513 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7feaec01c070 con 0x7feadc004af0 2024-09-21T03:26:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.660+0000 7feae2ffd640 1 -- 172.21.15.27:0/2644196513 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7feaec01d3b0 con 0x7feadc004af0 2024-09-21T03:26:54.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.660+0000 7feae0ff9640 1 -- 172.21.15.27:0/2644196513 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaf408c380 con 0x7feadc004af0 2024-09-21T03:26:54.666 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.665+0000 7feae2ffd640 1 -- 172.21.15.27:0/2644196513 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7feaec023030 con 0x7feadc004af0 2024-09-21T03:26:54.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.691+0000 7f475e656640 1 -- 172.21.15.27:0/2579815076 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f475001e100 msgr2=0x7f47500bf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.691+0000 7f475e656640 1 --2- 172.21.15.27:0/2579815076 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f475001e100 0x7f47500bf4a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4754005fd0 tx=0x7f475402a480 comp rx=0 tx=0).stop 2024-09-21T03:26:54.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.691+0000 7f475e656640 1 -- 172.21.15.27:0/2579815076 shutdown_connections 2024-09-21T03:26:54.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.691+0000 7f475e656640 1 --2- 172.21.15.27:0/2579815076 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f475001e100 0x7f47500bf4a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.691+0000 7f475e656640 1 --2- 172.21.15.27:0/2579815076 >> v2:172.21.15.27:3300/0 conn(0x7f475001dcf0 0x7f47500bcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.691+0000 7f475e656640 1 -- 172.21.15.27:0/2579815076 >> 172.21.15.27:0/2579815076 conn(0x7f4750005a10 msgr2=0x7f4750005e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.692+0000 7f475e656640 1 -- 172.21.15.27:0/2579815076 shutdown_connections 2024-09-21T03:26:54.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.697+0000 7f475e656640 1 -- 172.21.15.27:0/2579815076 wait complete. 2024-09-21T03:26:54.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.698+0000 7f475e656640 1 Processor -- start 2024-09-21T03:26:54.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.698+0000 7f475e656640 1 -- start start 2024-09-21T03:26:54.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.699+0000 7f475e656640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f475001c1f0 0x7f475000f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.699+0000 7f475e656640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f475001dcf0 0x7f475000fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.700 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.699+0000 7f475d654640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f475001c1f0 0x7f475000f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.699+0000 7f475e656640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4750010370 0x7f4750153920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.699+0000 7f475e656640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f47500c0450 con 0x7f475001dcf0 2024-09-21T03:26:54.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.699+0000 7f475e656640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f47500c02d0 con 0x7f4750010370 2024-09-21T03:26:54.701 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.699+0000 7f475e656640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f47500c05d0 con 0x7f475001c1f0 2024-09-21T03:26:54.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.699+0000 7f475d654640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f475001c1f0 0x7f475000f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:55484/0 (socket says 172.21.15.27:55484) 2024-09-21T03:26:54.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.699+0000 7f475d654640 1 -- 172.21.15.27:0/4174099254 learned_addr learned my addr 172.21.15.27:0/4174099254 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:54.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.700+0000 7f475d654640 1 -- 172.21.15.27:0/4174099254 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4750010370 msgr2=0x7f4750153920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.700+0000 7f475d654640 1 --2- 172.21.15.27:0/4174099254 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4750010370 0x7f4750153920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.700+0000 7f475d654640 1 -- 172.21.15.27:0/4174099254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f475001dcf0 msgr2=0x7f475000fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.700+0000 7f475d654640 1 --2- 172.21.15.27:0/4174099254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f475001dcf0 0x7f475000fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.701+0000 7f475d654640 1 -- 172.21.15.27:0/4174099254 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4750011af0 con 0x7f475001c1f0 2024-09-21T03:26:54.702 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.701+0000 7f475d654640 1 --2- 172.21.15.27:0/4174099254 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f475001c1f0 0x7f475000f8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f474c00b450 tx=0x7f474c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:54.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.702+0000 7f4749ffb640 1 -- 172.21.15.27:0/4174099254 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474c015020 con 0x7f475001c1f0 2024-09-21T03:26:54.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.703+0000 7f475e656640 1 -- 172.21.15.27:0/4174099254 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4750013080 con 0x7f475001c1f0 2024-09-21T03:26:54.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.703+0000 7f475e656640 1 -- 172.21.15.27:0/4174099254 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f4750010800 con 0x7f475001c1f0 2024-09-21T03:26:54.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.704+0000 7f4749ffb640 1 -- 172.21.15.27:0/4174099254 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f474c0119a0 con 0x7f475001c1f0 2024-09-21T03:26:54.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.704+0000 7f4749ffb640 1 -- 172.21.15.27:0/4174099254 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f474c011df0 con 0x7f475001c1f0 2024-09-21T03:26:54.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.704+0000 7f4749ffb640 1 -- 172.21.15.27:0/4174099254 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f474c018020 con 0x7f475001c1f0 2024-09-21T03:26:54.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.704+0000 7f4749ffb640 1 -- 172.21.15.27:0/4174099254 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f474c010a90 con 0x7f475001c1f0 2024-09-21T03:26:54.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.706+0000 7fd8c09db640 1 -- 172.21.15.27:0/451553255 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7fd8a0004360 con 0x7fd8b400cef0 2024-09-21T03:26:54.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.708+0000 7f475e656640 1 -- 172.21.15.27:0/4174099254 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4738005290 con 0x7f475001c1f0 2024-09-21T03:26:54.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.714+0000 7f4749ffb640 1 -- 172.21.15.27:0/4174099254 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f474c01f030 con 0x7f475001c1f0 2024-09-21T03:26:54.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.723+0000 7fd8aaffd640 1 -- 172.21.15.27:0/451553255 <== mon.1 v2:172.21.15.27:3301/0 12 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v9)=0 v9) ==== 115+0+0 (secure 0 0 0) 0x7fd8ac00f710 con 0x7fd8b400cef0 2024-09-21T03:26:54.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.730+0000 7fd8a8ff9640 1 -- 172.21.15.27:0/451553255 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd8b400cef0 msgr2=0x7fd8b401d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.730+0000 7fd8a8ff9640 1 --2- 172.21.15.27:0/451553255 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd8b400cef0 0x7fd8b401d2f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00ee70 tx=0x7fd8ac009670 comp rx=0 tx=0).stop 2024-09-21T03:26:54.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fd8a8ff9640 1 -- 172.21.15.27:0/451553255 shutdown_connections 2024-09-21T03:26:54.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fd8a8ff9640 1 --2- 172.21.15.27:0/451553255 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd8b40c9420 0x7fd8b401dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fd8a8ff9640 1 --2- 172.21.15.27:0/451553255 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd8b400dc10 0x7fd8b401d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fd8a8ff9640 1 --2- 172.21.15.27:0/451553255 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd8b400cef0 0x7fd8b401d2f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fd8a8ff9640 1 -- 172.21.15.27:0/451553255 >> 172.21.15.27:0/451553255 conn(0x7fd8b4005a30 msgr2=0x7fd8b400acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fd8a8ff9640 1 -- 172.21.15.27:0/451553255 shutdown_connections 2024-09-21T03:26:54.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fb03968e640 1 -- 172.21.15.27:0/1257074727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb03412a180 msgr2=0x7fb03412a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fb03968e640 1 --2- 172.21.15.27:0/1257074727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb03412a180 0x7fb03412a5c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb02800bc20 tx=0x7fb02802c2e0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fd8a8ff9640 1 -- 172.21.15.27:0/451553255 wait complete. 2024-09-21T03:26:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fb03968e640 1 -- 172.21.15.27:0/1257074727 shutdown_connections 2024-09-21T03:26:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fb03968e640 1 --2- 172.21.15.27:0/1257074727 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb03412a180 0x7fb03412a5c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fb03968e640 1 --2- 172.21.15.27:0/1257074727 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb0341270f0 0x7fb0341274d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.734+0000 7fb03968e640 1 -- 172.21.15.27:0/1257074727 >> 172.21.15.27:0/1257074727 conn(0x7fb03409a620 msgr2=0x7fb03409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.735+0000 7fb03968e640 1 -- 172.21.15.27:0/1257074727 shutdown_connections 2024-09-21T03:26:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.735+0000 7fb03968e640 1 -- 172.21.15.27:0/1257074727 wait complete. 2024-09-21T03:26:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.736+0000 7fb03968e640 1 Processor -- start 2024-09-21T03:26:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.736+0000 7fb03968e640 1 -- start start 2024-09-21T03:26:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.737+0000 7fb03968e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0341270f0 0x7fb03412c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.737+0000 7fb03968e640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb03412ab00 0x7fb03412c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.737+0000 7fb03968e640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb03412edd0 0x7fb03412cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.737+0000 7fb03968e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb034075550 con 0x7fb0341270f0 2024-09-21T03:26:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.737+0000 7fb03968e640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fb0340753d0 con 0x7fb03412edd0 2024-09-21T03:26:54.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.737+0000 7fb03968e640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fb0340756d0 con 0x7fb03412ab00 2024-09-21T03:26:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.737+0000 7fb038e8d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb03412edd0 0x7fb03412cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.737+0000 7fb038e8d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb03412edd0 0x7fb03412cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:41196/0 (socket says 172.21.15.27:41196) 2024-09-21T03:26:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.737+0000 7fb038e8d640 1 -- 172.21.15.27:0/2228229699 learned_addr learned my addr 172.21.15.27:0/2228229699 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.738+0000 7fb038e8d640 1 -- 172.21.15.27:0/2228229699 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb03412ab00 msgr2=0x7fb03412c9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:26:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.738+0000 7fb038e8d640 1 --2- 172.21.15.27:0/2228229699 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb03412ab00 0x7fb03412c9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.738+0000 7fb038e8d640 1 -- 172.21.15.27:0/2228229699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0341270f0 msgr2=0x7fb03412c460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:26:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.738+0000 7fb038e8d640 1 --2- 172.21.15.27:0/2228229699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0341270f0 0x7fb03412c460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.738+0000 7fb038e8d640 1 -- 172.21.15.27:0/2228229699 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03412f980 con 0x7fb03412edd0 2024-09-21T03:26:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.738+0000 7fb038e8d640 1 --2- 172.21.15.27:0/2228229699 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb03412edd0 0x7fb03412cee0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00cab0 tx=0x7fb02c00cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:54.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.740+0000 7fb030ff9640 1 -- 172.21.15.27:0/2228229699 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c017070 con 0x7fb03412edd0 2024-09-21T03:26:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.740+0000 7fb03968e640 1 -- 172.21.15.27:0/2228229699 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb03412e540 con 0x7fb03412edd0 2024-09-21T03:26:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.741+0000 7fb03968e640 1 -- 172.21.15.27:0/2228229699 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb03412d600 con 0x7fb03412edd0 2024-09-21T03:26:54.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.743+0000 7fb030ff9640 1 -- 172.21.15.27:0/2228229699 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb02c003c60 con 0x7fb03412edd0 2024-09-21T03:26:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.752+0000 7fb030ff9640 1 -- 172.21.15.27:0/2228229699 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c00aa80 con 0x7fb03412edd0 2024-09-21T03:26:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.752+0000 7fb030ff9640 1 -- 172.21.15.27:0/2228229699 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb02c01a070 con 0x7fb03412edd0 2024-09-21T03:26:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.752+0000 7fb030ff9640 1 -- 172.21.15.27:0/2228229699 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb02c01c780 con 0x7fb03412edd0 2024-09-21T03:26:54.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.753+0000 7fb01a7fc640 1 -- 172.21.15.27:0/2228229699 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb03412a180 con 0x7fb03412edd0 2024-09-21T03:26:54.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.757+0000 7fb030ff9640 1 -- 172.21.15.27:0/2228229699 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fb02c021020 con 0x7fb03412edd0 2024-09-21T03:26:54.769 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.768+0000 7fba9ed5e640 1 -- 172.21.15.27:0/1651893879 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fba78004360 con 0x7fba98095140 2024-09-21T03:26:54.776 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.775+0000 7fba957fa640 1 -- 172.21.15.27:0/1651893879 <== mon.2 v2:172.21.15.27:3302/0 12 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v10)=0 v10) ==== 106+0+0 (secure 0 0 0) 0x7fba8c00fdb0 con 0x7fba98095140 2024-09-21T03:26:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.781+0000 7fba9ed5e640 1 -- 172.21.15.27:0/1651893879 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fba98095140 msgr2=0x7fba981c7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.781+0000 7fba9ed5e640 1 --2- 172.21.15.27:0/1651893879 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fba98095140 0x7fba981c7190 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00c540 tx=0x7fba8c00ca00 comp rx=0 tx=0).stop 2024-09-21T03:26:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.781+0000 7fba9ed5e640 1 -- 172.21.15.27:0/1651893879 shutdown_connections 2024-09-21T03:26:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.781+0000 7fba9ed5e640 1 --2- 172.21.15.27:0/1651893879 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fba98095140 0x7fba981c7190 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.781+0000 7fba9ed5e640 1 --2- 172.21.15.27:0/1651893879 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba98094820 0x7fba98135100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.781+0000 7fba9ed5e640 1 --2- 172.21.15.27:0/1651893879 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fba98093d20 0x7fba98134bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.781+0000 7fba9ed5e640 1 -- 172.21.15.27:0/1651893879 >> 172.21.15.27:0/1651893879 conn(0x7fba9809c440 msgr2=0x7fba98133c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.782+0000 7fba9ed5e640 1 -- 172.21.15.27:0/1651893879 shutdown_connections 2024-09-21T03:26:54.783 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.782+0000 7fba9ed5e640 1 -- 172.21.15.27:0/1651893879 wait complete. 2024-09-21T03:26:54.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.790+0000 7f3db242b640 1 -- 172.21.15.27:0/488727628 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3dac094b90 msgr2=0x7f3dac12a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.790+0000 7f3daaffd640 1 -- 172.21.15.27:0/488727628 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da0031600 con 0x7f3dac094b90 2024-09-21T03:26:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.790+0000 7f3db242b640 1 --2- 172.21.15.27:0/488727628 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3dac094b90 0x7f3dac12a2e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009830 tx=0x7f3da002a3f0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.791+0000 7f3db242b640 1 -- 172.21.15.27:0/488727628 shutdown_connections 2024-09-21T03:26:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.791+0000 7f3db242b640 1 --2- 172.21.15.27:0/488727628 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3dac12a820 0x7f3dac12ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.791+0000 7f3db242b640 1 --2- 172.21.15.27:0/488727628 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3dac094b90 0x7f3dac12a2e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.791+0000 7f3db242b640 1 --2- 172.21.15.27:0/488727628 >> v2:172.21.15.27:3300/0 conn(0x7f3dac094270 0x7f3dac094650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.791+0000 7f3db242b640 1 -- 172.21.15.27:0/488727628 >> 172.21.15.27:0/488727628 conn(0x7f3dac09a100 msgr2=0x7f3dac09a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.791+0000 7f3db242b640 1 -- 172.21.15.27:0/488727628 shutdown_connections 2024-09-21T03:26:54.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.791+0000 7f3db242b640 1 -- 172.21.15.27:0/488727628 wait complete. 2024-09-21T03:26:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.793+0000 7f3db242b640 1 Processor -- start 2024-09-21T03:26:54.794 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.793+0000 7f3db242b640 1 -- start start 2024-09-21T03:26:54.795 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.794+0000 7f3db242b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3dac094270 0x7f3dac1c0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.795+0000 7f3dabfff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3dac094270 0x7f3dac1c0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.795+0000 7f3dabfff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3dac094270 0x7f3dac1c0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:55512/0 (socket says 172.21.15.27:55512) 2024-09-21T03:26:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.795+0000 7f3dabfff640 1 -- 172.21.15.27:0/4289996207 learned_addr learned my addr 172.21.15.27:0/4289996207 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.795+0000 7f3db242b640 1 --2- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dac12a820 0x7f3dac1c0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.796 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.795+0000 7f3db242b640 1 --2- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3dac08d950 0x7f3dac08af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.795+0000 7f3db09c8640 1 --2- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3dac08d950 0x7f3dac08af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.795+0000 7f3db242b640 1 -- 172.21.15.27:0/4289996207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac131eb0 con 0x7f3dac12a820 2024-09-21T03:26:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.795+0000 7f3db242b640 1 -- 172.21.15.27:0/4289996207 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3dac131d30 con 0x7f3dac08d950 2024-09-21T03:26:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.796+0000 7f3db242b640 1 -- 172.21.15.27:0/4289996207 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3dac132030 con 0x7f3dac094270 2024-09-21T03:26:54.797 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.796+0000 7f3db09c8640 1 -- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3dac094270 msgr2=0x7f3dac1c0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.796+0000 7f3db09c8640 1 --2- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3dac094270 0x7f3dac1c0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.796+0000 7f3db09c8640 1 -- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dac12a820 msgr2=0x7f3dac1c0a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:26:54.798 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.796+0000 7f3db09c8640 1 --2- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dac12a820 0x7f3dac1c0a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.796+0000 7f3db09c8640 1 -- 172.21.15.27:0/4289996207 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac08cf90 con 0x7f3dac08d950 2024-09-21T03:26:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.797+0000 7f3db09c8640 1 --2- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3dac08d950 0x7f3dac08af60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3da400eb40 tx=0x7f3da4013450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.797+0000 7f3dabfff640 1 --2- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3dac094270 0x7f3dac1c0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:26:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.797+0000 7f3da8ff9640 1 -- 172.21.15.27:0/4289996207 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da400e070 con 0x7f3dac08d950 2024-09-21T03:26:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.798+0000 7f3db242b640 1 -- 172.21.15.27:0/4289996207 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dac073890 con 0x7f3dac08d950 2024-09-21T03:26:54.799 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.798+0000 7f3db242b640 1 -- 172.21.15.27:0/4289996207 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dac093770 con 0x7f3dac08d950 2024-09-21T03:26:54.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.800+0000 7f3da8ff9640 1 -- 172.21.15.27:0/4289996207 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3da400e210 con 0x7f3dac08d950 2024-09-21T03:26:54.801 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.800+0000 7f3da8ff9640 1 -- 172.21.15.27:0/4289996207 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da4004d80 con 0x7f3dac08d950 2024-09-21T03:26:54.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.801+0000 7f3da8ff9640 1 -- 172.21.15.27:0/4289996207 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3da40053b0 con 0x7f3dac08d950 2024-09-21T03:26:54.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.801+0000 7f3da8ff9640 1 -- 172.21.15.27:0/4289996207 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3da4024730 con 0x7f3dac08d950 2024-09-21T03:26:54.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.802+0000 7f3d927fc640 1 -- 172.21.15.27:0/4289996207 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d8c005290 con 0x7f3dac08d950 2024-09-21T03:26:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.802+0000 7effcda8f640 1 -- 172.21.15.27:0/1465424087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8092a70 msgr2=0x7effc808a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.802+0000 7effcda8f640 1 --2- 172.21.15.27:0/1465424087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8092a70 0x7effc808a910 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7effb8009880 tx=0x7effb802a360 comp rx=0 tx=0).stop 2024-09-21T03:26:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.803+0000 7effcda8f640 1 -- 172.21.15.27:0/1465424087 shutdown_connections 2024-09-21T03:26:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.803+0000 7effcda8f640 1 --2- 172.21.15.27:0/1465424087 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7effc808b280 0x7effc8087830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.803+0000 7effcda8f640 1 --2- 172.21.15.27:0/1465424087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8092a70 0x7effc808a910 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.803+0000 7effcda8f640 1 -- 172.21.15.27:0/1465424087 >> 172.21.15.27:0/1465424087 conn(0x7effc809c0b0 msgr2=0x7effc809c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.806+0000 7f3da8ff9640 1 -- 172.21.15.27:0/4289996207 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f3da40056a0 con 0x7f3dac08d950 2024-09-21T03:26:54.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.803+0000 7effcda8f640 1 -- 172.21.15.27:0/1465424087 shutdown_connections 2024-09-21T03:26:54.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.803+0000 7effcda8f640 1 -- 172.21.15.27:0/1465424087 wait complete. 2024-09-21T03:26:54.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.804+0000 7effcda8f640 1 Processor -- start 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.805+0000 7effcda8f640 1 -- start start 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.805+0000 7effcda8f640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7effc808ae50 0x7effc8086010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.805+0000 7effcda8f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc808b280 0x7effc8086550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.805+0000 7effcda8f640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7effc8086a90 0x7effc808d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.805+0000 7effcda8f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7effc8143470 con 0x7effc808b280 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.805+0000 7effcda8f640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7effc81432f0 con 0x7effc808ae50 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.805+0000 7effcda8f640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7effc81435f0 con 0x7effc8086a90 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.806+0000 7effc6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc808b280 0x7effc8086550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.806+0000 7effc6ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc808b280 0x7effc8086550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:42988/0 (socket says 172.21.15.27:42988) 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.806+0000 7effc6ffd640 1 -- 172.21.15.27:0/4029464165 learned_addr learned my addr 172.21.15.27:0/4029464165 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.806+0000 7effc6ffd640 1 -- 172.21.15.27:0/4029464165 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7effc8086a90 msgr2=0x7effc808d040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.806+0000 7effc6ffd640 1 --2- 172.21.15.27:0/4029464165 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7effc8086a90 0x7effc808d040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.806+0000 7effc6ffd640 1 -- 172.21.15.27:0/4029464165 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7effc808ae50 msgr2=0x7effc8086010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:26:54.809 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.806+0000 7effc6ffd640 1 --2- 172.21.15.27:0/4029464165 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7effc808ae50 0x7effc8086010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.806+0000 7effc6ffd640 1 -- 172.21.15.27:0/4029464165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc808ebb0 con 0x7effc808b280 2024-09-21T03:26:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.807+0000 7effc6ffd640 1 --2- 172.21.15.27:0/4029464165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc808b280 0x7effc8086550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7effc000cbe0 tx=0x7effc000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:54.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.807+0000 7effaffff640 1 -- 172.21.15.27:0/4029464165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0017070 con 0x7effc808b280 2024-09-21T03:26:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.807+0000 7effcda8f640 1 -- 172.21.15.27:0/4029464165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc808fdb0 con 0x7effc808b280 2024-09-21T03:26:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.807+0000 7effcda8f640 1 -- 172.21.15.27:0/4029464165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc808d760 con 0x7effc808b280 2024-09-21T03:26:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.814+0000 7effaffff640 1 -- 172.21.15.27:0/4029464165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7effc000ac50 con 0x7effc808b280 2024-09-21T03:26:54.815 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.814+0000 7effaffff640 1 -- 172.21.15.27:0/4029464165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0003bd0 con 0x7effc808b280 2024-09-21T03:26:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.815+0000 7effaffff640 1 -- 172.21.15.27:0/4029464165 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7effc001a070 con 0x7effc808b280 2024-09-21T03:26:54.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.815+0000 7effaffff640 1 -- 172.21.15.27:0/4029464165 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7effc001c8f0 con 0x7effc808b280 2024-09-21T03:26:54.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.816+0000 7effcda8f640 1 -- 172.21.15.27:0/4029464165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effa8005290 con 0x7effc808b280 2024-09-21T03:26:54.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.820+0000 7effaffff640 1 -- 172.21.15.27:0/4029464165 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7effc0021020 con 0x7effc808b280 2024-09-21T03:26:54.833 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi027') 2024-09-21T03:26:54.834 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi027') 2024-09-21T03:26:54.850 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:54.849+0000 7f2af1337040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T03:26:54.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.850+0000 7f4c7ca2b640 1 -- 172.21.15.27:0/3004324095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c78095000 msgr2=0x7f4c78094630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.850+0000 7f4c7ca2b640 1 --2- 172.21.15.27:0/3004324095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c78095000 0x7f4c78094630 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4c70005fd0 tx=0x7f4c7002a9c0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.850+0000 7f4c7ca2b640 1 -- 172.21.15.27:0/3004324095 shutdown_connections 2024-09-21T03:26:54.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.850+0000 7f4c7ca2b640 1 --2- 172.21.15.27:0/3004324095 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4c78090090 0x7f4c78092550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.850+0000 7f4c7ca2b640 1 --2- 172.21.15.27:0/3004324095 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c78095000 0x7f4c78094630 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.852 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.850+0000 7f4c7ca2b640 1 -- 172.21.15.27:0/3004324095 >> 172.21.15.27:0/3004324095 conn(0x7f4c7809c440 msgr2=0x7f4c7809c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.851+0000 7f4c7ca2b640 1 -- 172.21.15.27:0/3004324095 shutdown_connections 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.851+0000 7f4c7ca2b640 1 -- 172.21.15.27:0/3004324095 wait complete. 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.852+0000 7f4c7ca2b640 1 Processor -- start 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.852+0000 7f4c7ca2b640 1 -- start start 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.853+0000 7f4c7ca2b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4c78090090 0x7f4c781e02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.853+0000 7f4c7ca2b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f4c78093b40 0x7f4c781e0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.853+0000 7f4c7ca2b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c78146c80 0x7f4c78143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.853+0000 7f4c7ca2b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4c78143810 con 0x7f4c78146c80 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.853+0000 7f4c7ca2b640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f4c78143690 con 0x7f4c78090090 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.853+0000 7f4c7ca2b640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f4c78143990 con 0x7f4c78093b40 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.853+0000 7f4c7659c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4c78090090 0x7f4c781e02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.853+0000 7f4c7659c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4c78090090 0x7f4c781e02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:41248/0 (socket says 172.21.15.27:41248) 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.853+0000 7f4c7659c640 1 -- 172.21.15.27:0/434440468 learned_addr learned my addr 172.21.15.27:0/434440468 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:54.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.854+0000 7f4c7659c640 1 -- 172.21.15.27:0/434440468 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f4c78093b40 msgr2=0x7f4c781e0830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:26:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.854+0000 7f4c7659c640 1 --2- 172.21.15.27:0/434440468 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f4c78093b40 0x7f4c781e0830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.854+0000 7f4c7659c640 1 -- 172.21.15.27:0/434440468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c78146c80 msgr2=0x7f4c78143f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:26:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.854+0000 7f4c7659c640 1 --2- 172.21.15.27:0/434440468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c78146c80 0x7f4c78143f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.854+0000 7f4c7659c640 1 -- 172.21.15.27:0/434440468 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c781462f0 con 0x7f4c78090090 2024-09-21T03:26:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.854+0000 7f4c7659c640 1 --2- 172.21.15.27:0/434440468 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4c78090090 0x7f4c781e02f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800c750 tx=0x7f4c6800cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.854+0000 7f4c66ffd640 1 -- 172.21.15.27:0/434440468 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c680050d0 con 0x7f4c78090090 2024-09-21T03:26:54.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.855+0000 7f4c7ca2b640 1 -- 172.21.15.27:0/434440468 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c781476f0 con 0x7f4c78090090 2024-09-21T03:26:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.855+0000 7f4c7ca2b640 1 -- 172.21.15.27:0/434440468 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c78144660 con 0x7f4c78090090 2024-09-21T03:26:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.856+0000 7f4c66ffd640 1 -- 172.21.15.27:0/434440468 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4c68005270 con 0x7f4c78090090 2024-09-21T03:26:54.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.856+0000 7f4c66ffd640 1 -- 172.21.15.27:0/434440468 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6800a640 con 0x7f4c78090090 2024-09-21T03:26:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.858+0000 7f4c7ca2b640 1 -- 172.21.15.27:0/434440468 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c5c005290 con 0x7f4c78090090 2024-09-21T03:26:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.858+0000 7f8d42c24640 1 -- 172.21.15.27:0/4026478277 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d34014290 msgr2=0x7f8d3400fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.858+0000 7f8d42c24640 1 --2- 172.21.15.27:0/4026478277 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d34014290 0x7f8d3400fdb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c0748d0 tx=0x7f8d3c074930 comp rx=0 tx=0).stop 2024-09-21T03:26:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.858+0000 7f8d42c24640 1 -- 172.21.15.27:0/4026478277 shutdown_connections 2024-09-21T03:26:54.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.858+0000 7f8d42c24640 1 --2- 172.21.15.27:0/4026478277 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8d34014950 0x7f8d340102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.858+0000 7f8d42c24640 1 --2- 172.21.15.27:0/4026478277 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d34014290 0x7f8d3400fdb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.858+0000 7f8d42c24640 1 -- 172.21.15.27:0/4026478277 >> 172.21.15.27:0/4026478277 conn(0x7f8d340226b0 msgr2=0x7f8d34022ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.858+0000 7f8d42c24640 1 -- 172.21.15.27:0/4026478277 shutdown_connections 2024-09-21T03:26:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.858+0000 7f8d42c24640 1 -- 172.21.15.27:0/4026478277 wait complete. 2024-09-21T03:26:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.859+0000 7f8d42c24640 1 Processor -- start 2024-09-21T03:26:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.860+0000 7f8d42c24640 1 -- start start 2024-09-21T03:26:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.860+0000 7f8d42c24640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d34010830 0x7f8d340b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.860+0000 7f8d42c24640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d34014950 0x7f8d340b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.860+0000 7f8d42c24640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8d340bb7f0 0x7f8d340b9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.860+0000 7f8d42c24640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d340c0470 con 0x7f8d34014950 2024-09-21T03:26:54.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.860+0000 7f8d42c24640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f8d340c02f0 con 0x7f8d34010830 2024-09-21T03:26:54.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.860+0000 7f8d42c24640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f8d340c05f0 con 0x7f8d340bb7f0 2024-09-21T03:26:54.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.861+0000 7f8d3bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d34014950 0x7f8d340b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.861+0000 7f8d3bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d34014950 0x7f8d340b8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43018/0 (socket says 172.21.15.27:43018) 2024-09-21T03:26:54.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.861+0000 7f8d3bfff640 1 -- 172.21.15.27:0/1630036016 learned_addr learned my addr 172.21.15.27:0/1630036016 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:54.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.861+0000 7f8d3bfff640 1 -- 172.21.15.27:0/1630036016 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8d340bb7f0 msgr2=0x7f8d340b9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.861+0000 7f8d3bfff640 1 --2- 172.21.15.27:0/1630036016 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8d340bb7f0 0x7f8d340b9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.861+0000 7f8d3bfff640 1 -- 172.21.15.27:0/1630036016 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d34010830 msgr2=0x7f8d340b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.861+0000 7f8d3bfff640 1 --2- 172.21.15.27:0/1630036016 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d34010830 0x7f8d340b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.861+0000 7f8d3bfff640 1 -- 172.21.15.27:0/1630036016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d340bc840 con 0x7f8d34014950 2024-09-21T03:26:54.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.862+0000 7f8d3bfff640 1 --2- 172.21.15.27:0/1630036016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d34014950 0x7f8d340b8bf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800b470 tx=0x7f8d2800b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:54.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.862+0000 7f8d397fa640 1 -- 172.21.15.27:0/1630036016 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28015020 con 0x7f8d34014950 2024-09-21T03:26:54.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.862+0000 7f8d42c24640 1 -- 172.21.15.27:0/1630036016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d3400f430 con 0x7f8d34014950 2024-09-21T03:26:54.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.862+0000 7f8d42c24640 1 -- 172.21.15.27:0/1630036016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d340b9850 con 0x7f8d34014950 2024-09-21T03:26:54.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.864+0000 7f8d397fa640 1 -- 172.21.15.27:0/1630036016 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d28011980 con 0x7f8d34014950 2024-09-21T03:26:54.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.864+0000 7f8d397fa640 1 -- 172.21.15.27:0/1630036016 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28011db0 con 0x7f8d34014950 2024-09-21T03:26:54.865 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:54.864+0000 7f3c11c12040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T03:26:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.869+0000 7f8d26ffd640 1 -- 172.21.15.27:0/1630036016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d1c005290 con 0x7f8d34014950 2024-09-21T03:26:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.869+0000 7f8d397fa640 1 -- 172.21.15.27:0/1630036016 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8d28018020 con 0x7f8d34014950 2024-09-21T03:26:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.869+0000 7f8d397fa640 1 -- 172.21.15.27:0/1630036016 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8d28010a40 con 0x7f8d34014950 2024-09-21T03:26:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.871+0000 7f4c66ffd640 1 -- 172.21.15.27:0/434440468 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4c6800a820 con 0x7f4c78090090 2024-09-21T03:26:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.871+0000 7f4c66ffd640 1 -- 172.21.15.27:0/434440468 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4c68012760 con 0x7f4c78090090 2024-09-21T03:26:54.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.871+0000 7f4c66ffd640 1 -- 172.21.15.27:0/434440468 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f4c781462f0 con 0x7f4c78090090 2024-09-21T03:26:54.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.873+0000 7f8d397fa640 1 -- 172.21.15.27:0/1630036016 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f8d2801f020 con 0x7f8d34014950 2024-09-21T03:26:54.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.878+0000 7f0762e6e640 1 -- 172.21.15.27:0/1733165379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0936e0 msgr2=0x7f075c093ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.878+0000 7f0762e6e640 1 --2- 172.21.15.27:0/1733165379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0936e0 0x7f075c093ac0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0758009830 tx=0x7f075802a320 comp rx=0 tx=0).stop 2024-09-21T03:26:54.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.878+0000 7f0762e6e640 1 -- 172.21.15.27:0/1733165379 shutdown_connections 2024-09-21T03:26:54.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.878+0000 7f0762e6e640 1 --2- 172.21.15.27:0/1733165379 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f075c094470 0x7f075c08aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.878+0000 7f0762e6e640 1 --2- 172.21.15.27:0/1733165379 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c0936e0 0x7f075c093ac0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.878+0000 7f0762e6e640 1 -- 172.21.15.27:0/1733165379 >> 172.21.15.27:0/1733165379 conn(0x7f075c09c1a0 msgr2=0x7f075c09c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.878+0000 7f0762e6e640 1 -- 172.21.15.27:0/1733165379 shutdown_connections 2024-09-21T03:26:54.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.878+0000 7f0762e6e640 1 -- 172.21.15.27:0/1733165379 wait complete. 2024-09-21T03:26:54.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.880+0000 7f0762e6e640 1 Processor -- start 2024-09-21T03:26:54.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.880+0000 7f0762e6e640 1 -- start start 2024-09-21T03:26:54.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.880+0000 7f0762e6e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c094000 0x7f075c070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.880+0000 7f0762e6e640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f075c094470 0x7f075c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.880+0000 7f0762e6e640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f075c0821e0 0x7f075c0715f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.880+0000 7f0762e6e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f075c143350 con 0x7f075c094000 2024-09-21T03:26:54.885 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.880+0000 7f0762e6e640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f075c1431d0 con 0x7f075c0821e0 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.880+0000 7f0762e6e640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f075c1434d0 con 0x7f075c094470 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.881+0000 7f076166b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f075c094470 0x7f075c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.881+0000 7f076166b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f075c094470 0x7f075c0710b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:55530/0 (socket says 172.21.15.27:55530) 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.881+0000 7f076166b640 1 -- 172.21.15.27:0/2898578756 learned_addr learned my addr 172.21.15.27:0/2898578756 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.881+0000 7f0761e6c640 1 --2- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c094000 0x7f075c070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.881+0000 7f0761e6c640 1 -- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f075c094470 msgr2=0x7f075c0710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.881+0000 7f0761e6c640 1 --2- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f075c094470 0x7f075c0710b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.881+0000 7f0761e6c640 1 -- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f075c0821e0 msgr2=0x7f075c0715f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.881+0000 7f0761e6c640 1 --2- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f075c0821e0 0x7f075c0715f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.881+0000 7f0761e6c640 1 -- 172.21.15.27:0/2898578756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075c0726a0 con 0x7f075c094000 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.882+0000 7f0761e6c640 1 --2- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c094000 0x7f075c070b70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0758031000 tx=0x7f07580043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:54.886 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.883+0000 7f07527fc640 1 -- 172.21.15.27:0/2898578756 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758037050 con 0x7f075c094000 2024-09-21T03:26:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.883+0000 7f0762e6e640 1 -- 172.21.15.27:0/2898578756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f075c08c090 con 0x7f075c094000 2024-09-21T03:26:54.887 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.884+0000 7f0762e6e640 1 -- 172.21.15.27:0/2898578756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f075c072d30 con 0x7f075c094000 2024-09-21T03:26:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.885+0000 7f076166b640 1 --2- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f075c094470 0x7f075c0710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:26:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.885+0000 7f07527fc640 1 -- 172.21.15.27:0/2898578756 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0758038040 con 0x7f075c094000 2024-09-21T03:26:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.885+0000 7f07527fc640 1 -- 172.21.15.27:0/2898578756 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0758003850 con 0x7f075c094000 2024-09-21T03:26:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.886+0000 7f07527fc640 1 -- 172.21.15.27:0/2898578756 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0758045020 con 0x7f075c094000 2024-09-21T03:26:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.886+0000 7f07527fc640 1 -- 172.21.15.27:0/2898578756 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f075803e5d0 con 0x7f075c094000 2024-09-21T03:26:54.888 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.887+0000 7f0762e6e640 1 -- 172.21.15.27:0/2898578756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0744005290 con 0x7f075c094000 2024-09-21T03:26:54.892 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.891+0000 7f07527fc640 1 -- 172.21.15.27:0/2898578756 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f075804c020 con 0x7f075c094000 2024-09-21T03:26:54.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.901+0000 7fb31139c640 1 -- 172.21.15.27:0/3947168960 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb30c12ca90 msgr2=0x7fb30c12ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.901+0000 7fb31139c640 1 --2- 172.21.15.27:0/3947168960 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb30c12ca90 0x7fb30c12ce90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4009930 tx=0x7fb2f402a480 comp rx=0 tx=0).stop 2024-09-21T03:26:54.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.906+0000 7fb31139c640 1 -- 172.21.15.27:0/3947168960 shutdown_connections 2024-09-21T03:26:54.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.906+0000 7fb31139c640 1 --2- 172.21.15.27:0/3947168960 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb30c12d3d0 0x7fb30c12d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.906+0000 7fb31139c640 1 --2- 172.21.15.27:0/3947168960 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb30c12ca90 0x7fb30c12ce90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.911 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.906+0000 7fb31139c640 1 -- 172.21.15.27:0/3947168960 >> 172.21.15.27:0/3947168960 conn(0x7fb30c09c930 msgr2=0x7fb30c09cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.918+0000 7fb31139c640 1 -- 172.21.15.27:0/3947168960 shutdown_connections 2024-09-21T03:26:54.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.922+0000 7fb31139c640 1 -- 172.21.15.27:0/3947168960 wait complete. 2024-09-21T03:26:54.925 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.923+0000 7fb31139c640 1 Processor -- start 2024-09-21T03:26:54.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.923+0000 7fb31139c640 1 -- start start 2024-09-21T03:26:54.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.924+0000 7fb31139c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb30c072d70 0x7fb30c1bfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.924+0000 7fb31139c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb30c12ca90 0x7fb30c1c0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.924+0000 7fb31139c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb30c12d3d0 0x7fb30c1c0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.924+0000 7fb31139c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb30c12e670 con 0x7fb30c12ca90 2024-09-21T03:26:54.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.924+0000 7fb31139c640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fb30c12e4f0 con 0x7fb30c072d70 2024-09-21T03:26:54.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.924+0000 7fb31139c640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fb30c12e7f0 con 0x7fb30c12d3d0 2024-09-21T03:26:54.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.925+0000 7fb30affd640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb30c072d70 0x7fb30c1bfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.925+0000 7fb30affd640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb30c072d70 0x7fb30c1bfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:41266/0 (socket says 172.21.15.27:41266) 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.925+0000 7fb30affd640 1 -- 172.21.15.27:0/223807486 learned_addr learned my addr 172.21.15.27:0/223807486 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.925+0000 7fb30b7fe640 1 --2- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb30c12d3d0 0x7fb30c1c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.926+0000 7fb30a7fc640 1 --2- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb30c12ca90 0x7fb30c1c0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.926+0000 7fb30affd640 1 -- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb30c12d3d0 msgr2=0x7fb30c1c0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.926+0000 7fb30affd640 1 --2- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb30c12d3d0 0x7fb30c1c0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.926+0000 7fb30affd640 1 -- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb30c12ca90 msgr2=0x7fb30c1c0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.926+0000 7fb30affd640 1 --2- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb30c12ca90 0x7fb30c1c0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.926+0000 7fb30affd640 1 -- 172.21.15.27:0/223807486 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb30c12b240 con 0x7fb30c072d70 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.927+0000 7fb30affd640 1 --2- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb30c072d70 0x7fb30c1bfb00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb30400b700 tx=0x7fb30400bbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.927+0000 7fb3037fe640 1 -- 172.21.15.27:0/223807486 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304019070 con 0x7fb30c072d70 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.927+0000 7fb3037fe640 1 -- 172.21.15.27:0/223807486 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb30400c8f0 con 0x7fb30c072d70 2024-09-21T03:26:54.929 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.927+0000 7fb3037fe640 1 -- 172.21.15.27:0/223807486 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30400cd40 con 0x7fb30c072d70 2024-09-21T03:26:54.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.927+0000 7fb31139c640 1 -- 172.21.15.27:0/223807486 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb30c12bad0 con 0x7fb30c072d70 2024-09-21T03:26:54.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.928+0000 7fb31139c640 1 -- 172.21.15.27:0/223807486 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb30c08af60 con 0x7fb30c072d70 2024-09-21T03:26:54.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.928+0000 7fb3037fe640 1 -- 172.21.15.27:0/223807486 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb30401c070 con 0x7fb30c072d70 2024-09-21T03:26:54.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.928+0000 7fb3037fe640 1 -- 172.21.15.27:0/223807486 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb3040148b0 con 0x7fb30c072d70 2024-09-21T03:26:54.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.929+0000 7f3a59255640 1 -- 172.21.15.27:0/2134008857 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a4c0bd080 msgr2=0x7f3a4c0bf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.929+0000 7f3a59255640 1 --2- 172.21.15.27:0/2134008857 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a4c0bd080 0x7f3a4c0bf470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a54069e40 tx=0x7f3a54074930 comp rx=0 tx=0).stop 2024-09-21T03:26:54.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.930+0000 7fb31139c640 1 -- 172.21.15.27:0/223807486 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f0005290 con 0x7fb30c072d70 2024-09-21T03:26:54.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.934+0000 7fb3037fe640 1 -- 172.21.15.27:0/223807486 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fb304023070 con 0x7fb30c072d70 2024-09-21T03:26:54.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.941+0000 7f3a59255640 1 -- 172.21.15.27:0/2134008857 shutdown_connections 2024-09-21T03:26:54.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.941+0000 7f3a59255640 1 --2- 172.21.15.27:0/2134008857 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a4c0bd080 0x7f3a4c0bf470 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.941+0000 7f3a59255640 1 --2- 172.21.15.27:0/2134008857 >> v2:172.21.15.27:3300/0 conn(0x7f3a4c014950 0x7f3a4c0104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.942 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.941+0000 7f3a59255640 1 -- 172.21.15.27:0/2134008857 >> 172.21.15.27:0/2134008857 conn(0x7f3a4c0226b0 msgr2=0x7f3a4c022ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.943 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.942+0000 7f3a59255640 1 -- 172.21.15.27:0/2134008857 shutdown_connections 2024-09-21T03:26:54.944 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.942+0000 7f3a59255640 1 -- 172.21.15.27:0/2134008857 wait complete. 2024-09-21T03:26:54.945 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.943+0000 7f3a59255640 1 Processor -- start 2024-09-21T03:26:54.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.944+0000 7f3a59255640 1 -- start start 2024-09-21T03:26:54.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3a59255640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a4c014290 0x7f3a4c0b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3a59255640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a4c014950 0x7f3a4c0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3a59255640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a4c0bd080 0x7f3a4c01e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3a59255640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c0c0390 con 0x7f3a4c014290 2024-09-21T03:26:54.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3a59255640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3a4c0c0210 con 0x7f3a4c0bd080 2024-09-21T03:26:54.946 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3a59255640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3a4c0c0510 con 0x7f3a4c014950 2024-09-21T03:26:54.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3a58a54640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a4c0bd080 0x7f3a4c01e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3a58a54640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a4c0bd080 0x7f3a4c01e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:41282/0 (socket says 172.21.15.27:41282) 2024-09-21T03:26:54.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3a58a54640 1 -- 172.21.15.27:0/3016111518 learned_addr learned my addr 172.21.15.27:0/3016111518 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:54.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3f6993a640 1 -- 172.21.15.27:0/1435716128 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3f5c00e4d0 msgr2=0x7f3f5c0af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3f6993a640 1 --2- 172.21.15.27:0/1435716128 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3f5c00e4d0 0x7f3f5c0af6c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3f58005fd0 tx=0x7f3f5802a660 comp rx=0 tx=0).stop 2024-09-21T03:26:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3a537fe640 1 --2- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a4c014950 0x7f3a4c0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.945+0000 7f3a53fff640 1 --2- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a4c014290 0x7f3a4c0b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3f6993a640 1 -- 172.21.15.27:0/1435716128 shutdown_connections 2024-09-21T03:26:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3f6993a640 1 --2- 172.21.15.27:0/1435716128 >> v2:172.21.15.27:3300/0 conn(0x7f3f5c0afc00 0x7f3f5c0b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3f6993a640 1 --2- 172.21.15.27:0/1435716128 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3f5c00e4d0 0x7f3f5c0af6c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3f6993a640 1 --2- 172.21.15.27:0/1435716128 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3f5c00dbb0 0x7f3f5c00df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.948 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3f6993a640 1 -- 172.21.15.27:0/1435716128 >> 172.21.15.27:0/1435716128 conn(0x7f3f5c005a10 msgr2=0x7f3f5c005e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3f6993a640 1 -- 172.21.15.27:0/1435716128 shutdown_connections 2024-09-21T03:26:54.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3a537fe640 1 -- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a4c0bd080 msgr2=0x7f3a4c01e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3a537fe640 1 --2- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a4c0bd080 0x7f3a4c01e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3a537fe640 1 -- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a4c014290 msgr2=0x7f3a4c0b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3a537fe640 1 --2- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a4c014290 0x7f3a4c0b38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3a537fe640 1 -- 172.21.15.27:0/3016111518 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c01e0b0 con 0x7f3a4c014950 2024-09-21T03:26:54.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3f6993a640 1 -- 172.21.15.27:0/1435716128 wait complete. 2024-09-21T03:26:54.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.946+0000 7f3a537fe640 1 --2- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a4c014950 0x7f3a4c0b3e30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000e9c0 tx=0x7f3a4000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:54.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.947+0000 7f3a50ff9640 1 -- 172.21.15.27:0/3016111518 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4000ccc0 con 0x7f3a4c014950 2024-09-21T03:26:54.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.947+0000 7f3a59255640 1 -- 172.21.15.27:0/3016111518 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a4c01d040 con 0x7f3a4c014950 2024-09-21T03:26:54.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.947+0000 7f3a59255640 1 -- 172.21.15.27:0/3016111518 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a4c01da80 con 0x7f3a4c014950 2024-09-21T03:26:54.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.948+0000 7f3f6993a640 1 Processor -- start 2024-09-21T03:26:54.951 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.949+0000 7f3a50ff9640 1 -- 172.21.15.27:0/3016111518 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a40010430 con 0x7f3a4c014950 2024-09-21T03:26:54.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.949+0000 7f3a50ff9640 1 -- 172.21.15.27:0/3016111518 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a40010860 con 0x7f3a4c014950 2024-09-21T03:26:54.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.949+0000 7f3a50ff9640 1 -- 172.21.15.27:0/3016111518 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3a40010a00 con 0x7f3a4c014950 2024-09-21T03:26:54.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.949+0000 7f3a50ff9640 1 -- 172.21.15.27:0/3016111518 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3a400213f0 con 0x7f3a4c014950 2024-09-21T03:26:54.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.949+0000 7f3a3e7fc640 1 -- 172.21.15.27:0/3016111518 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a4c00dc60 con 0x7f3a4c014950 2024-09-21T03:26:54.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.950+0000 7f3f6993a640 1 -- start start 2024-09-21T03:26:54.952 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.952+0000 7f3f6993a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f5c00dbb0 0x7f3f5c145120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.952+0000 7f3f6993a640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3f5c00e4d0 0x7f3f5c145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.953 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.952+0000 7f3f6993a640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3f5c0afc00 0x7f3f5c0a8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:54.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.957+0000 7f3f63fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3f5c00e4d0 0x7f3f5c145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.957+0000 7f3f63fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3f5c00e4d0 0x7f3f5c145660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:41294/0 (socket says 172.21.15.27:41294) 2024-09-21T03:26:54.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.957+0000 7f3f68938640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f5c00dbb0 0x7f3f5c145120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.957+0000 7f3f68938640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f5c00dbb0 0x7f3f5c145120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43062/0 (socket says 172.21.15.27:43062) 2024-09-21T03:26:54.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.957+0000 7f3f6993a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5c0b6b70 con 0x7f3f5c00dbb0 2024-09-21T03:26:54.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.957+0000 7f3f6993a640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3f5c0b69f0 con 0x7f3f5c00e4d0 2024-09-21T03:26:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.957+0000 7f3f63fff640 1 -- 172.21.15.27:0/4079297792 learned_addr learned my addr 172.21.15.27:0/4079297792 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.961+0000 7f3f69139640 1 --2- 172.21.15.27:0/4079297792 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3f5c0afc00 0x7f3f5c0a8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.961+0000 7f3f6993a640 1 -- 172.21.15.27:0/4079297792 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3f5c0b6cf0 con 0x7f3f5c0afc00 2024-09-21T03:26:54.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.961+0000 7f3a50ff9640 1 -- 172.21.15.27:0/3016111518 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f3a4c00dc60 con 0x7f3a4c014950 2024-09-21T03:26:54.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.962+0000 7f3f69139640 1 -- 172.21.15.27:0/4079297792 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3f5c00e4d0 msgr2=0x7f3f5c145660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.962+0000 7f3f69139640 1 --2- 172.21.15.27:0/4079297792 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3f5c00e4d0 0x7f3f5c145660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.962+0000 7f3f69139640 1 -- 172.21.15.27:0/4079297792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f5c00dbb0 msgr2=0x7f3f5c145120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.962+0000 7f3f69139640 1 --2- 172.21.15.27:0/4079297792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f5c00dbb0 0x7f3f5c145120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.962+0000 7f3f69139640 1 -- 172.21.15.27:0/4079297792 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5c145ba0 con 0x7f3f5c0afc00 2024-09-21T03:26:54.964 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.958+0000 7feae0ff9640 1 -- 172.21.15.27:0/2644196513 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7feaf4075050 con 0x7feadc004af0 2024-09-21T03:26:54.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.964+0000 7f3f69139640 1 --2- 172.21.15.27:0/4079297792 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3f5c0afc00 0x7f3f5c0a8500 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3f50012b50 tx=0x7f3f50010490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:54.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.964+0000 7f3f617fa640 1 -- 172.21.15.27:0/4079297792 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5000d0d0 con 0x7f3f5c0afc00 2024-09-21T03:26:54.965 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.964+0000 7f3f617fa640 1 -- 172.21.15.27:0/4079297792 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3f5000d270 con 0x7f3f5c0afc00 2024-09-21T03:26:54.966 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.964+0000 7f3f617fa640 1 -- 172.21.15.27:0/4079297792 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f50010d20 con 0x7f3f5c0afc00 2024-09-21T03:26:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.966+0000 7f3f6993a640 1 -- 172.21.15.27:0/4079297792 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f5c0ab110 con 0x7f3f5c0afc00 2024-09-21T03:26:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.966+0000 7f3f6993a640 1 -- 172.21.15.27:0/4079297792 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f5c0aa710 con 0x7f3f5c0afc00 2024-09-21T03:26:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.966+0000 7f3f617fa640 1 -- 172.21.15.27:0/4079297792 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3f5001f030 con 0x7f3f5c0afc00 2024-09-21T03:26:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.970+0000 7f3f6993a640 1 -- 172.21.15.27:0/4079297792 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f5c0abda0 con 0x7f3f5c0afc00 2024-09-21T03:26:54.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.972+0000 7feae2ffd640 1 -- 172.21.15.27:0/2644196513 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v11)=0 v11) ==== 122+0+0 (secure 0 0 0) 0x7feaec01e440 con 0x7feadc004af0 2024-09-21T03:26:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.975+0000 7f3f617fa640 1 -- 172.21.15.27:0/4079297792 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3f50005360 con 0x7f3f5c0afc00 2024-09-21T03:26:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.975+0000 7feae0ff9640 1 -- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7feadc004af0 msgr2=0x7feaf41250b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.975+0000 7feae0ff9640 1 --2- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7feadc004af0 0x7feaf41250b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feaf406df70 tx=0x7feaec009ed0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.975+0000 7feae0ff9640 1 -- 172.21.15.27:0/2644196513 shutdown_connections 2024-09-21T03:26:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.975+0000 7feae0ff9640 1 --2- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7feaf412a560 0x7feaf4125b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.975+0000 7feae0ff9640 1 --2- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7feadc006500 0x7feaf41255f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.975+0000 7feae0ff9640 1 --2- 172.21.15.27:0/2644196513 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7feadc004af0 0x7feaf41250b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.975+0000 7feae0ff9640 1 -- 172.21.15.27:0/2644196513 >> 172.21.15.27:0/2644196513 conn(0x7feaf409c930 msgr2=0x7feaf4072050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.975+0000 7feae0ff9640 1 -- 172.21.15.27:0/2644196513 shutdown_connections 2024-09-21T03:26:54.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.975+0000 7feae0ff9640 1 -- 172.21.15.27:0/2644196513 wait complete. 2024-09-21T03:26:54.983 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:54.979+0000 7f3f617fa640 1 -- 172.21.15.27:0/4079297792 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f3f5c0abda0 con 0x7f3f5c0afc00 2024-09-21T03:26:55.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.000+0000 7ff93359e640 1 -- 172.21.15.27:0/3890004713 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff92c01c1f0 msgr2=0x7ff92c01c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.000+0000 7ff93359e640 1 --2- 172.21.15.27:0/3890004713 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff92c01c1f0 0x7ff92c01c5d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff928009830 tx=0x7ff92802a300 comp rx=0 tx=0).stop 2024-09-21T03:26:55.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.001+0000 7ff93359e640 1 -- 172.21.15.27:0/3890004713 shutdown_connections 2024-09-21T03:26:55.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.001+0000 7ff93359e640 1 --2- 172.21.15.27:0/3890004713 >> v2:172.21.15.27:3300/0 conn(0x7ff92c01d600 0x7ff92c01da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.001+0000 7ff93359e640 1 --2- 172.21.15.27:0/3890004713 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff92c01c1f0 0x7ff92c01c5d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.001+0000 7ff93359e640 1 -- 172.21.15.27:0/3890004713 >> 172.21.15.27:0/3890004713 conn(0x7ff92c005a10 msgr2=0x7ff92c005e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.004 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.004+0000 7ff93359e640 1 -- 172.21.15.27:0/3890004713 shutdown_connections 2024-09-21T03:26:55.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.004+0000 7ff93359e640 1 -- 172.21.15.27:0/3890004713 wait complete. 2024-09-21T03:26:55.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.005+0000 7ff93359e640 1 Processor -- start 2024-09-21T03:26:55.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.005+0000 7ff93359e640 1 -- start start 2024-09-21T03:26:55.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.006+0000 7ff93359e640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff92c01c1f0 0x7ff92c0b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.006+0000 7ff93359e640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff92c01cb10 0x7ff92c0b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.006+0000 7ff93359e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c01d600 0x7ff92c00f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.006+0000 7ff93359e640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c0c0490 con 0x7ff92c01d600 2024-09-21T03:26:55.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.006+0000 7ff93359e640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7ff92c0c0310 con 0x7ff92c01cb10 2024-09-21T03:26:55.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.006+0000 7ff93359e640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7ff92c0c0610 con 0x7ff92c01c1f0 2024-09-21T03:26:55.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.006+0000 7ff932d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c01d600 0x7ff92c00f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.006+0000 7ff932d9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c01d600 0x7ff92c00f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43086/0 (socket says 172.21.15.27:43086) 2024-09-21T03:26:55.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.007+0000 7ff932d9d640 1 -- 172.21.15.27:0/2614186619 learned_addr learned my addr 172.21.15.27:0/2614186619 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:55.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.007+0000 7ff93259c640 1 --2- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff92c01c1f0 0x7ff92c0b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.007+0000 7ff931d9b640 1 --2- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff92c01cb10 0x7ff92c0b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.007+0000 7ff932d9d640 1 -- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff92c01c1f0 msgr2=0x7ff92c0b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.007+0000 7ff932d9d640 1 --2- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff92c01c1f0 0x7ff92c0b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.007+0000 7ff932d9d640 1 -- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff92c01cb10 msgr2=0x7ff92c0b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.007+0000 7ff932d9d640 1 --2- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff92c01cb10 0x7ff92c0b3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.007+0000 7ff932d9d640 1 -- 172.21.15.27:0/2614186619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c0112e0 con 0x7ff92c01d600 2024-09-21T03:26:55.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.008+0000 7ff932d9d640 1 --2- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c01d600 0x7ff92c00f840 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff934069aa0 tx=0x7ff934068ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:55.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.016+0000 7ff922ffd640 1 -- 172.21.15.27:0/2614186619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934067db0 con 0x7ff92c01d600 2024-09-21T03:26:55.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.016+0000 7ff922ffd640 1 -- 172.21.15.27:0/2614186619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff934069220 con 0x7ff92c01d600 2024-09-21T03:26:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.016+0000 7ff93359e640 1 -- 172.21.15.27:0/2614186619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff92c0123c0 con 0x7ff92c01d600 2024-09-21T03:26:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.016+0000 7ff93359e640 1 -- 172.21.15.27:0/2614186619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff92c00fd80 con 0x7ff92c01d600 2024-09-21T03:26:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.017+0000 7ff93359e640 1 -- 172.21.15.27:0/2614186619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff92c01c5d0 con 0x7ff92c01d600 2024-09-21T03:26:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.021+0000 7ff922ffd640 1 -- 172.21.15.27:0/2614186619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934069470 con 0x7ff92c01d600 2024-09-21T03:26:55.023 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.017+0000 7f3d927fc640 1 -- 172.21.15.27:0/4289996207 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f3d8c005850 con 0x7f3dac08d950 2024-09-21T03:26:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.021+0000 7f42a7fff640 1 -- 172.21.15.27:0/1277653384 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f42a8094000 msgr2=0x7f42a808a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.021+0000 7f42a7fff640 1 --2- 172.21.15.27:0/1277653384 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f42a8094000 0x7f42a808a7d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f429c005b90 tx=0x7f429c02b550 comp rx=0 tx=0).stop 2024-09-21T03:26:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.021+0000 7f42a7fff640 1 -- 172.21.15.27:0/1277653384 shutdown_connections 2024-09-21T03:26:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.021+0000 7f42a7fff640 1 --2- 172.21.15.27:0/1277653384 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f42a8094000 0x7f42a808a7d0 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f429c005b90 tx=0x7f429c02b550 comp rx=0 tx=0).stop 2024-09-21T03:26:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.021+0000 7f42a7fff640 1 --2- 172.21.15.27:0/1277653384 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f42a80936e0 0x7f42a8093ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.021+0000 7f42a7fff640 1 -- 172.21.15.27:0/1277653384 >> 172.21.15.27:0/1277653384 conn(0x7f42a809c1a0 msgr2=0x7f42a809c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.022+0000 7ff922ffd640 1 -- 172.21.15.27:0/2614186619 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff9340753e0 con 0x7ff92c01d600 2024-09-21T03:26:55.024 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.022+0000 7ff922ffd640 1 -- 172.21.15.27:0/2614186619 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff9340686d0 con 0x7ff92c01d600 2024-09-21T03:26:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.022+0000 7ff922ffd640 1 -- 172.21.15.27:0/2614186619 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7ff934094480 con 0x7ff92c01d600 2024-09-21T03:26:55.025 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.022+0000 7f42a7fff640 1 -- 172.21.15.27:0/1277653384 shutdown_connections 2024-09-21T03:26:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.024+0000 7f42a7fff640 1 -- 172.21.15.27:0/1277653384 wait complete. 2024-09-21T03:26:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.025+0000 7f42a7fff640 1 Processor -- start 2024-09-21T03:26:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.026+0000 7f42a7fff640 1 -- start start 2024-09-21T03:26:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.026+0000 7f42a7fff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f42a80936e0 0x7f42a80848f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.026+0000 7f42a7fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f42a8094440 0x7f42a808b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.026+0000 7f42a7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42a808d770 0x7f42a808be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.026+0000 7f42a6ffd640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f42a80936e0 0x7f42a80848f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.026+0000 7f42a6ffd640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f42a80936e0 0x7f42a80848f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:55622/0 (socket says 172.21.15.27:55622) 2024-09-21T03:26:55.032 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.027+0000 7f42a67fc640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f42a8094440 0x7f42a808b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.027+0000 7f42a67fc640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f42a8094440 0x7f42a808b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:41322/0 (socket says 172.21.15.27:41322) 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.027+0000 7f42a77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42a808d770 0x7f42a808be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.027+0000 7f42a77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42a808d770 0x7f42a808be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43102/0 (socket says 172.21.15.27:43102) 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.027+0000 7f42a6ffd640 1 -- 172.21.15.27:0/772415234 learned_addr learned my addr 172.21.15.27:0/772415234 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.027+0000 7f42a7fff640 1 -- 172.21.15.27:0/772415234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f42a814bbb0 con 0x7f42a808d770 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.027+0000 7f42a7fff640 1 -- 172.21.15.27:0/772415234 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f42a814ba30 con 0x7f42a8094440 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.027+0000 7f42a7fff640 1 -- 172.21.15.27:0/772415234 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f42a814bd30 con 0x7f42a80936e0 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.028+0000 7f42a77fe640 1 -- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f42a80936e0 msgr2=0x7f42a80848f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.028+0000 7f42a77fe640 1 --2- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f42a80936e0 0x7f42a80848f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.028+0000 7f42a77fe640 1 -- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f42a8094440 msgr2=0x7f42a808b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.028+0000 7f42a77fe640 1 --2- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f42a8094440 0x7f42a808b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.028+0000 7f42a77fe640 1 -- 172.21.15.27:0/772415234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a8073150 con 0x7f42a808d770 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.028+0000 7f42a6ffd640 1 --2- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f42a80936e0 0x7f42a80848f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:26:55.033 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.028+0000 7f42a77fe640 1 --2- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42a808d770 0x7f42a808be20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f42a000cac0 tx=0x7f42a000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.028+0000 7f42a67fc640 1 --2- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f42a8094440 0x7f42a808b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:26:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.033+0000 7f475e656640 1 -- 172.21.15.27:0/4174099254 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7f4738005580 con 0x7f475001c1f0 2024-09-21T03:26:55.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.033+0000 7f3da8ff9640 1 -- 172.21.15.27:0/4289996207 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v12)=0 v12) ==== 121+0+0 (secure 0 0 0) 0x7f3da4005880 con 0x7f3dac08d950 2024-09-21T03:26:55.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.034+0000 7f428f7fe640 1 -- 172.21.15.27:0/772415234 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a0017070 con 0x7f42a808d770 2024-09-21T03:26:55.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.034+0000 7f42a7fff640 1 -- 172.21.15.27:0/772415234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a808cb30 con 0x7f42a808d770 2024-09-21T03:26:55.037 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.034+0000 7f42a7fff640 1 -- 172.21.15.27:0/772415234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a808df50 con 0x7f42a808d770 2024-09-21T03:26:55.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.035+0000 7f3db242b640 1 -- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3dac08d950 msgr2=0x7f3dac08af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.035+0000 7f3db242b640 1 --2- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3dac08d950 0x7f3dac08af60 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3da400eb40 tx=0x7f3da4013450 comp rx=0 tx=0).stop 2024-09-21T03:26:55.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.035+0000 7f3db242b640 1 -- 172.21.15.27:0/4289996207 shutdown_connections 2024-09-21T03:26:55.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.035+0000 7f3db242b640 1 --2- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3dac08d950 0x7f3dac08af60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.035+0000 7f3db242b640 1 --2- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3dac12a820 0x7f3dac1c0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.035+0000 7f3db242b640 1 --2- 172.21.15.27:0/4289996207 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3dac094270 0x7f3dac1c0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.035+0000 7f3db242b640 1 -- 172.21.15.27:0/4289996207 >> 172.21.15.27:0/4289996207 conn(0x7f3dac09a100 msgr2=0x7f3dac071da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.038 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.035+0000 7f3db242b640 1 -- 172.21.15.27:0/4289996207 shutdown_connections 2024-09-21T03:26:55.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.035+0000 7f428f7fe640 1 -- 172.21.15.27:0/772415234 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f42a0003c60 con 0x7f42a808d770 2024-09-21T03:26:55.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.036+0000 7f428f7fe640 1 -- 172.21.15.27:0/772415234 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a00052d0 con 0x7f42a808d770 2024-09-21T03:26:55.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.036+0000 7f428f7fe640 1 -- 172.21.15.27:0/772415234 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f42a001a070 con 0x7f42a808d770 2024-09-21T03:26:55.039 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.036+0000 7f428f7fe640 1 -- 172.21.15.27:0/772415234 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f42a001c780 con 0x7f42a808d770 2024-09-21T03:26:55.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.039+0000 7f3db242b640 1 -- 172.21.15.27:0/4289996207 wait complete. 2024-09-21T03:26:55.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.039+0000 7f428d7fa640 1 -- 172.21.15.27:0/772415234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4288005290 con 0x7f42a808d770 2024-09-21T03:26:55.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.057+0000 7f428f7fe640 1 -- 172.21.15.27:0/772415234 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f42a000ad60 con 0x7f42a808d770 2024-09-21T03:26:55.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.057+0000 7f4749ffb640 1 -- 172.21.15.27:0/4174099254 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v13)=0 v13) ==== 121+0+0 (secure 0 0 0) 0x7f4750011af0 con 0x7f475001c1f0 2024-09-21T03:26:55.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.061+0000 7f475e656640 1 -- 172.21.15.27:0/4174099254 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f475001c1f0 msgr2=0x7f475000f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.061+0000 7f475e656640 1 --2- 172.21.15.27:0/4174099254 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f475001c1f0 0x7f475000f8f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f474c00b450 tx=0x7f474c00b910 comp rx=0 tx=0).stop 2024-09-21T03:26:55.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.061+0000 7f475e656640 1 -- 172.21.15.27:0/4174099254 shutdown_connections 2024-09-21T03:26:55.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.061+0000 7f475e656640 1 --2- 172.21.15.27:0/4174099254 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4750010370 0x7f4750153920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.061+0000 7f475e656640 1 --2- 172.21.15.27:0/4174099254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f475001dcf0 0x7f475000fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.061+0000 7f475e656640 1 --2- 172.21.15.27:0/4174099254 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f475001c1f0 0x7f475000f8f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.061+0000 7f475e656640 1 -- 172.21.15.27:0/4174099254 >> 172.21.15.27:0/4174099254 conn(0x7f4750005a10 msgr2=0x7f47500bde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.061+0000 7f475e656640 1 -- 172.21.15.27:0/4174099254 shutdown_connections 2024-09-21T03:26:55.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.061+0000 7f475e656640 1 -- 172.21.15.27:0/4174099254 wait complete. 2024-09-21T03:26:55.063 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.061+0000 7fb01a7fc640 1 -- 172.21.15.27:0/2228229699 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7fb0340758e0 con 0x7fb03412edd0 2024-09-21T03:26:55.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.075+0000 7fb030ff9640 1 -- 172.21.15.27:0/2228229699 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v14)=0 v14) ==== 119+0+0 (secure 0 0 0) 0x7fb02c00ad00 con 0x7fb03412edd0 2024-09-21T03:26:55.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.077+0000 7fb01a7fc640 1 -- 172.21.15.27:0/2228229699 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb03412edd0 msgr2=0x7fb03412cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.077+0000 7fb01a7fc640 1 --2- 172.21.15.27:0/2228229699 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb03412edd0 0x7fb03412cee0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00cab0 tx=0x7fb02c00cf70 comp rx=0 tx=0).stop 2024-09-21T03:26:55.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.077+0000 7fb01a7fc640 1 -- 172.21.15.27:0/2228229699 shutdown_connections 2024-09-21T03:26:55.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.077+0000 7fb01a7fc640 1 --2- 172.21.15.27:0/2228229699 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb03412edd0 0x7fb03412cee0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.077+0000 7fb01a7fc640 1 --2- 172.21.15.27:0/2228229699 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb03412ab00 0x7fb03412c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.077+0000 7fb01a7fc640 1 --2- 172.21.15.27:0/2228229699 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb0341270f0 0x7fb03412c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.077+0000 7fb01a7fc640 1 -- 172.21.15.27:0/2228229699 >> 172.21.15.27:0/2228229699 conn(0x7fb03409a620 msgr2=0x7fb034097a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.077+0000 7fb01a7fc640 1 -- 172.21.15.27:0/2228229699 shutdown_connections 2024-09-21T03:26:55.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.078+0000 7fb01a7fc640 1 -- 172.21.15.27:0/2228229699 wait complete. 2024-09-21T03:26:55.079 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.078+0000 7effcda8f640 1 -- 172.21.15.27:0/4029464165 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7effa8005580 con 0x7effc808b280 2024-09-21T03:26:55.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.084+0000 7f8d26ffd640 1 -- 172.21.15.27:0/1630036016 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f8d1c005850 con 0x7f8d34014950 2024-09-21T03:26:55.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.087+0000 7effaffff640 1 -- 172.21.15.27:0/4029464165 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v15)=0 v15) ==== 120+0+0 (secure 0 0 0) 0x7effc001cb40 con 0x7effc808b280 2024-09-21T03:26:55.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.089+0000 7effadffb640 1 -- 172.21.15.27:0/4029464165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc808b280 msgr2=0x7effc8086550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.089+0000 7effadffb640 1 --2- 172.21.15.27:0/4029464165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc808b280 0x7effc8086550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7effc000cbe0 tx=0x7effc000a430 comp rx=0 tx=0).stop 2024-09-21T03:26:55.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.089+0000 7effadffb640 1 -- 172.21.15.27:0/4029464165 shutdown_connections 2024-09-21T03:26:55.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.089+0000 7effadffb640 1 --2- 172.21.15.27:0/4029464165 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7effc8086a90 0x7effc808d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.089+0000 7effadffb640 1 --2- 172.21.15.27:0/4029464165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc808b280 0x7effc8086550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.089+0000 7effadffb640 1 --2- 172.21.15.27:0/4029464165 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7effc808ae50 0x7effc8086010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.089+0000 7effadffb640 1 -- 172.21.15.27:0/4029464165 >> 172.21.15.27:0/4029464165 conn(0x7effc809c0b0 msgr2=0x7effc8098260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.089+0000 7effadffb640 1 -- 172.21.15.27:0/4029464165 shutdown_connections 2024-09-21T03:26:55.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.090+0000 7effadffb640 1 -- 172.21.15.27:0/4029464165 wait complete. 2024-09-21T03:26:55.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.094+0000 7f4c7ca2b640 1 -- 172.21.15.27:0/434440468 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command([{prefix=config set, name=debug_mgr}] v 0) -- 0x7f4c5c005850 con 0x7f4c78090090 2024-09-21T03:26:55.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.099+0000 7f8d397fa640 1 -- 172.21.15.27:0/1630036016 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v16)=0 v16) ==== 120+0+0 (secure 0 0 0) 0x7f8d2800d040 con 0x7f8d34014950 2024-09-21T03:26:55.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.100+0000 7f8d26ffd640 1 -- 172.21.15.27:0/1630036016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d34014950 msgr2=0x7f8d340b8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.100+0000 7f8d26ffd640 1 --2- 172.21.15.27:0/1630036016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d34014950 0x7f8d340b8bf0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800b470 tx=0x7f8d2800b930 comp rx=0 tx=0).stop 2024-09-21T03:26:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.101+0000 7f8d26ffd640 1 -- 172.21.15.27:0/1630036016 shutdown_connections 2024-09-21T03:26:55.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.101+0000 7f8d26ffd640 1 --2- 172.21.15.27:0/1630036016 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8d340bb7f0 0x7f8d340b9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.101+0000 7f8d26ffd640 1 --2- 172.21.15.27:0/1630036016 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d34014950 0x7f8d340b8bf0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.101+0000 7f8d26ffd640 1 --2- 172.21.15.27:0/1630036016 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d34010830 0x7f8d340b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.101+0000 7f8d26ffd640 1 -- 172.21.15.27:0/1630036016 >> 172.21.15.27:0/1630036016 conn(0x7f8d340226b0 msgr2=0x7f8d34012150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.101+0000 7f8d26ffd640 1 -- 172.21.15.27:0/1630036016 shutdown_connections 2024-09-21T03:26:55.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.101+0000 7f8d26ffd640 1 -- 172.21.15.27:0/1630036016 wait complete. 2024-09-21T03:26:55.104 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.108 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.108 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.116+0000 7f4c66ffd640 1 -- 172.21.15.27:0/434440468 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mgr}]=0 v17)=0 v17) ==== 108+0+0 (secure 0 0 0) 0x7f4c6800abe0 con 0x7f4c78090090 2024-09-21T03:26:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.117+0000 7f4c64ff9640 1 -- 172.21.15.27:0/434440468 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4c78090090 msgr2=0x7f4c781e02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.117+0000 7f4c64ff9640 1 --2- 172.21.15.27:0/434440468 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4c78090090 0x7f4c781e02f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4c6800c750 tx=0x7f4c6800cc10 comp rx=0 tx=0).stop 2024-09-21T03:26:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.117+0000 7f4c64ff9640 1 -- 172.21.15.27:0/434440468 shutdown_connections 2024-09-21T03:26:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.117+0000 7f4c64ff9640 1 --2- 172.21.15.27:0/434440468 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c78146c80 0x7f4c78143f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.117+0000 7f4c64ff9640 1 --2- 172.21.15.27:0/434440468 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f4c78093b40 0x7f4c781e0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.117+0000 7f4c64ff9640 1 --2- 172.21.15.27:0/434440468 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4c78090090 0x7f4c781e02f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.117+0000 7f4c64ff9640 1 -- 172.21.15.27:0/434440468 >> 172.21.15.27:0/434440468 conn(0x7f4c7809c440 msgr2=0x7f4c78096a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.117+0000 7f4c64ff9640 1 -- 172.21.15.27:0/434440468 shutdown_connections 2024-09-21T03:26:55.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.118+0000 7f4c64ff9640 1 -- 172.21.15.27:0/434440468 wait complete. 2024-09-21T03:26:55.122 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.131 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'client_mount_timeout', '30'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.132+0000 7f0762e6e640 1 -- 172.21.15.27:0/2898578756 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f0744005580 con 0x7f075c094000 2024-09-21T03:26:55.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.142+0000 7f07527fc640 1 -- 172.21.15.27:0/2898578756 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v18)=0 v18) ==== 106+0+0 (secure 0 0 0) 0x7f0758032630 con 0x7f075c094000 2024-09-21T03:26:55.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.163+0000 7f0762e6e640 1 -- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c094000 msgr2=0x7f075c070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.163+0000 7f0762e6e640 1 --2- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c094000 0x7f075c070b70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0758031000 tx=0x7f07580043b0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.163+0000 7f0762e6e640 1 -- 172.21.15.27:0/2898578756 shutdown_connections 2024-09-21T03:26:55.164 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.163+0000 7f0762e6e640 1 --2- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f075c0821e0 0x7f075c0715f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.163+0000 7f0762e6e640 1 --2- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f075c094470 0x7f075c0710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.163+0000 7f0762e6e640 1 --2- 172.21.15.27:0/2898578756 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f075c094000 0x7f075c070b70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.163+0000 7f0762e6e640 1 -- 172.21.15.27:0/2898578756 >> 172.21.15.27:0/2898578756 conn(0x7f075c09c1a0 msgr2=0x7f075c098790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.163+0000 7f0762e6e640 1 -- 172.21.15.27:0/2898578756 shutdown_connections 2024-09-21T03:26:55.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.163+0000 7f0762e6e640 1 -- 172.21.15.27:0/2898578756 wait complete. 2024-09-21T03:26:55.201 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:55.200+0000 7f2af1337040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T03:26:55.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.204+0000 7f3f6993a640 1 -- 172.21.15.27:0/4079297792 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7f3f5c022480 con 0x7f3f5c0afc00 2024-09-21T03:26:55.207 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:55.206+0000 7f3c11c12040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T03:26:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.214+0000 7f3f617fa640 1 -- 172.21.15.27:0/4079297792 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v19)=0 v19) ==== 125+0+0 (secure 0 0 0) 0x7f3f50026060 con 0x7f3f5c0afc00 2024-09-21T03:26:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.215+0000 7f3a3e7fc640 1 -- 172.21.15.27:0/3016111518 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7f3a4c00fdb0 con 0x7f3a4c014950 2024-09-21T03:26:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.216+0000 7f3f6993a640 1 -- 172.21.15.27:0/4079297792 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3f5c0afc00 msgr2=0x7f3f5c0a8500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.216+0000 7f3f6993a640 1 --2- 172.21.15.27:0/4079297792 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3f5c0afc00 0x7f3f5c0a8500 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3f50012b50 tx=0x7f3f50010490 comp rx=0 tx=0).stop 2024-09-21T03:26:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.216+0000 7f3f6993a640 1 -- 172.21.15.27:0/4079297792 shutdown_connections 2024-09-21T03:26:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.216+0000 7f3f6993a640 1 --2- 172.21.15.27:0/4079297792 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3f5c0afc00 0x7f3f5c0a8500 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.216+0000 7f3f6993a640 1 --2- 172.21.15.27:0/4079297792 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3f5c00e4d0 0x7f3f5c145660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.216+0000 7f3f6993a640 1 --2- 172.21.15.27:0/4079297792 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3f5c00dbb0 0x7f3f5c145120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.216+0000 7f3f6993a640 1 -- 172.21.15.27:0/4079297792 >> 172.21.15.27:0/4079297792 conn(0x7f3f5c005a10 msgr2=0x7f3f5c00ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.216+0000 7f3f6993a640 1 -- 172.21.15.27:0/4079297792 shutdown_connections 2024-09-21T03:26:55.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.216+0000 7f3f6993a640 1 -- 172.21.15.27:0/4079297792 wait complete. 2024-09-21T03:26:55.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.223+0000 7fb31139c640 1 -- 172.21.15.27:0/223807486 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7fb2f0005580 con 0x7fb30c072d70 2024-09-21T03:26:55.228 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.227+0000 7f3a50ff9640 1 -- 172.21.15.27:0/3016111518 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v20)=0 v20) ==== 119+0+0 (secure 0 0 0) 0x7f3a40018420 con 0x7f3a4c014950 2024-09-21T03:26:55.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.229+0000 7f3a3e7fc640 1 -- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a4c014950 msgr2=0x7f3a4c0b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.229+0000 7f3a3e7fc640 1 --2- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a4c014950 0x7f3a4c0b3e30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3a4000e9c0 tx=0x7f3a4000ee80 comp rx=0 tx=0).stop 2024-09-21T03:26:55.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.231+0000 7f3a3e7fc640 1 -- 172.21.15.27:0/3016111518 shutdown_connections 2024-09-21T03:26:55.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.231+0000 7f3a3e7fc640 1 --2- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a4c0bd080 0x7f3a4c01e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.231+0000 7f3a3e7fc640 1 --2- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a4c014950 0x7f3a4c0b3e30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.232 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.231+0000 7f3a3e7fc640 1 --2- 172.21.15.27:0/3016111518 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a4c014290 0x7f3a4c0b38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.231+0000 7f3a3e7fc640 1 -- 172.21.15.27:0/3016111518 >> 172.21.15.27:0/3016111518 conn(0x7f3a4c0226b0 msgr2=0x7f3a4c00cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.231+0000 7f3a3e7fc640 1 -- 172.21.15.27:0/3016111518 shutdown_connections 2024-09-21T03:26:55.233 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.231+0000 7f3a3e7fc640 1 -- 172.21.15.27:0/3016111518 wait complete. 2024-09-21T03:26:55.238 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.237+0000 7fb3037fe640 1 -- 172.21.15.27:0/223807486 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v21)=0 v21) ==== 111+0+0 (secure 0 0 0) 0x7fb304014bd0 con 0x7fb30c072d70 2024-09-21T03:26:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.240+0000 7fb31139c640 1 -- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb30c072d70 msgr2=0x7fb30c1bfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.240+0000 7fb31139c640 1 --2- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb30c072d70 0x7fb30c1bfb00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb30400b700 tx=0x7fb30400bbc0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.240+0000 7fb31139c640 1 -- 172.21.15.27:0/223807486 shutdown_connections 2024-09-21T03:26:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.240+0000 7fb31139c640 1 --2- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb30c12d3d0 0x7fb30c1c0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.240+0000 7fb31139c640 1 --2- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb30c12ca90 0x7fb30c1c0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.240+0000 7fb31139c640 1 --2- 172.21.15.27:0/223807486 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb30c072d70 0x7fb30c1bfb00 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb30400b700 tx=0x7fb30400bbc0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.240+0000 7fb31139c640 1 -- 172.21.15.27:0/223807486 >> 172.21.15.27:0/223807486 conn(0x7fb30c09c930 msgr2=0x7fb30c071350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.241 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.240+0000 7fb31139c640 1 -- 172.21.15.27:0/223807486 shutdown_connections 2024-09-21T03:26:55.242 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.240+0000 7fb31139c640 1 -- 172.21.15.27:0/223807486 wait complete. 2024-09-21T03:26:55.243 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.242+0000 7ff93359e640 1 -- 172.21.15.27:0/2614186619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7ff92c0008d0 con 0x7ff92c01d600 2024-09-21T03:26:55.249 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_client', '20'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.250 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.248+0000 7ff922ffd640 1 -- 172.21.15.27:0/2614186619 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v22)=0 v22) ==== 121+0+0 (secure 0 0 0) 0x7ff934069610 con 0x7ff92c01d600 2024-09-21T03:26:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.249+0000 7ff93359e640 1 -- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c01d600 msgr2=0x7ff92c00f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.249+0000 7ff93359e640 1 --2- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c01d600 0x7ff92c00f840 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff934069aa0 tx=0x7ff934068ad0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.249+0000 7ff93359e640 1 -- 172.21.15.27:0/2614186619 shutdown_connections 2024-09-21T03:26:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.249+0000 7ff93359e640 1 --2- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff92c01d600 0x7ff92c00f840 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.249+0000 7ff93359e640 1 --2- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff92c01cb10 0x7ff92c0b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.249+0000 7ff93359e640 1 --2- 172.21.15.27:0/2614186619 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff92c01c1f0 0x7ff92c0b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.249+0000 7ff93359e640 1 -- 172.21.15.27:0/2614186619 >> 172.21.15.27:0/2614186619 conn(0x7ff92c005a10 msgr2=0x7ff92c019490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.249+0000 7ff93359e640 1 -- 172.21.15.27:0/2614186619 shutdown_connections 2024-09-21T03:26:55.251 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.249+0000 7ff93359e640 1 -- 172.21.15.27:0/2614186619 wait complete. 2024-09-21T03:26:55.257 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_mgr', '20'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.257 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'debug_ms', '1'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.257 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mgr', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.257 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.255+0000 7f428d7fa640 1 -- 172.21.15.27:0/772415234 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f42880058c0 con 0x7f42a808d770 2024-09-21T03:26:55.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.258+0000 7f428f7fe640 1 -- 172.21.15.27:0/772415234 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v23)=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f42a000a680 con 0x7f42a808d770 2024-09-21T03:26:55.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.260+0000 7f428d7fa640 1 -- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42a808d770 msgr2=0x7f42a808be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.260+0000 7f428d7fa640 1 --2- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42a808d770 0x7f42a808be20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f42a000cac0 tx=0x7f42a000cf80 comp rx=0 tx=0).stop 2024-09-21T03:26:55.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.260+0000 7f428d7fa640 1 -- 172.21.15.27:0/772415234 shutdown_connections 2024-09-21T03:26:55.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.260+0000 7f428d7fa640 1 --2- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42a808d770 0x7f42a808be20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.260+0000 7f428d7fa640 1 --2- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f42a8094440 0x7f42a808b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.261+0000 7f428d7fa640 1 --2- 172.21.15.27:0/772415234 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f42a80936e0 0x7f42a80848f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.261+0000 7f428d7fa640 1 -- 172.21.15.27:0/772415234 >> 172.21.15.27:0/772415234 conn(0x7f42a809c1a0 msgr2=0x7f42a8087760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.261+0000 7f428d7fa640 1 -- 172.21.15.27:0/772415234 shutdown_connections 2024-09-21T03:26:55.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.261+0000 7f428d7fa640 1 -- 172.21.15.27:0/772415234 wait complete. 2024-09-21T03:26:55.264 DEBUG:tasks.ceph:waiting for RemoteProcess(client=, args=['sudo', 'ceph', '--cluster', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi027') 2024-09-21T03:26:55.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config dump 2024-09-21T03:26:55.347 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:55.346+0000 7f3c11c12040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T03:26:55.347 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:55.346+0000 7f2af1337040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T03:26:55.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.422+0000 7fbc52867640 1 -- 172.21.15.27:0/1547753759 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbc4c096ad0 msgr2=0x7fbc4c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.422+0000 7fbc52867640 1 --2- 172.21.15.27:0/1547753759 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbc4c096ad0 0x7fbc4c06e3e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c009860 tx=0x7fbc3c02a410 comp rx=0 tx=0).stop 2024-09-21T03:26:55.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.422+0000 7fbc52867640 1 -- 172.21.15.27:0/1547753759 shutdown_connections 2024-09-21T03:26:55.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.422+0000 7fbc52867640 1 --2- 172.21.15.27:0/1547753759 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbc4c1329e0 0x7fbc4c134dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.422+0000 7fbc52867640 1 --2- 172.21.15.27:0/1547753759 >> v2:172.21.15.27:3300/0 conn(0x7fbc4c06e920 0x7fbc4c06ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.422+0000 7fbc52867640 1 --2- 172.21.15.27:0/1547753759 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbc4c096ad0 0x7fbc4c06e3e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.422+0000 7fbc52867640 1 -- 172.21.15.27:0/1547753759 >> 172.21.15.27:0/1547753759 conn(0x7fbc4c099510 msgr2=0x7fbc4c099920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.422+0000 7fbc52867640 1 -- 172.21.15.27:0/1547753759 shutdown_connections 2024-09-21T03:26:55.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.423+0000 7fbc52867640 1 -- 172.21.15.27:0/1547753759 wait complete. 2024-09-21T03:26:55.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.424+0000 7fbc52867640 1 Processor -- start 2024-09-21T03:26:55.425 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.424+0000 7fbc52867640 1 -- start start 2024-09-21T03:26:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc52867640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbc4c06e920 0x7fbc4c1218f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc52867640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c096ad0 0x7fbc4c121e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc52867640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbc4c1329e0 0x7fbc4c122370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc4bfff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbc4c06e920 0x7fbc4c1218f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc52867640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c135860 con 0x7fbc4c096ad0 2024-09-21T03:26:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc52867640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4c1356e0 con 0x7fbc4c06e920 2024-09-21T03:26:55.427 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc4b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c096ad0 0x7fbc4c121e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc4bfff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbc4c06e920 0x7fbc4c1218f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:41344/0 (socket says 172.21.15.27:41344) 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc4bfff640 1 -- 172.21.15.27:0/936539666 learned_addr learned my addr 172.21.15.27:0/936539666 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc52867640 1 -- 172.21.15.27:0/936539666 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fbc4c1359e0 con 0x7fbc4c1329e0 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc50e04640 1 --2- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbc4c1329e0 0x7fbc4c122370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc50e04640 1 -- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbc4c06e920 msgr2=0x7fbc4c1218f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc50e04640 1 --2- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbc4c06e920 0x7fbc4c1218f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc50e04640 1 -- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c096ad0 msgr2=0x7fbc4c121e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc50e04640 1 --2- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c096ad0 0x7fbc4c121e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc50e04640 1 -- 172.21.15.27:0/936539666 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c096110 con 0x7fbc4c1329e0 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc4b7fe640 1 --2- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c096ad0 0x7fbc4c121e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.425+0000 7fbc4bfff640 1 --2- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbc4c06e920 0x7fbc4c1218f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.426+0000 7fbc50e04640 1 --2- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbc4c1329e0 0x7fbc4c122370 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbc44005fd0 tx=0x7fbc4400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.426+0000 7fbc48ff9640 1 -- 172.21.15.27:0/936539666 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc4400cd80 con 0x7fbc4c1329e0 2024-09-21T03:26:55.428 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.426+0000 7fbc48ff9640 1 -- 172.21.15.27:0/936539666 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbc44004c10 con 0x7fbc4c1329e0 2024-09-21T03:26:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.426+0000 7fbc52867640 1 -- 172.21.15.27:0/936539666 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4c122eb0 con 0x7fbc4c1329e0 2024-09-21T03:26:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.426+0000 7fbc48ff9640 1 -- 172.21.15.27:0/936539666 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc44005020 con 0x7fbc4c1329e0 2024-09-21T03:26:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.426+0000 7fbc48ff9640 1 -- 172.21.15.27:0/936539666 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbc44005550 con 0x7fbc4c1329e0 2024-09-21T03:26:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.427+0000 7fbc52867640 1 -- 172.21.15.27:0/936539666 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4c124170 con 0x7fbc4c1329e0 2024-09-21T03:26:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.427+0000 7fbc48ff9640 1 -- 172.21.15.27:0/936539666 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbc440187b0 con 0x7fbc4c1329e0 2024-09-21T03:26:55.429 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.427+0000 7fbc52867640 1 -- 172.21.15.27:0/936539666 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc4c06ed60 con 0x7fbc4c1329e0 2024-09-21T03:26:55.430 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.429+0000 7fbc48ff9640 1 -- 172.21.15.27:0/936539666 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fbc44027080 con 0x7fbc4c1329e0 2024-09-21T03:26:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.491+0000 7fbc52867640 1 -- 172.21.15.27:0/936539666 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7fbc4c067d30 con 0x7fbc4c1329e0 2024-09-21T03:26:55.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.491+0000 7fbc48ff9640 1 -- 172.21.15.27:0/936539666 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v23) ==== 59+0+1472 (secure 0 0 0) 0x7fbc440189b0 con 0x7fbc4c1329e0 2024-09-21T03:26:55.492 INFO:teuthology.orchestra.run.smithi027.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:mon advanced mon_op_complaint_time 120 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:mgr advanced client_mount_timeout 30 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:mgr advanced debug_client 20/20 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:mgr advanced debug_mgr 20/20 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:mgr advanced debug_ms 1/1 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:mds advanced debug_mds 20/20 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:mds advanced debug_mds_balancer 20/20 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:mds advanced debug_ms 2/2 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:mds dev mds_debug_frag true 2024-09-21T03:26:55.493 INFO:teuthology.orchestra.run.smithi027.stdout:mds dev mds_debug_scatterstat true 2024-09-21T03:26:55.494 INFO:teuthology.orchestra.run.smithi027.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-21T03:26:55.494 INFO:teuthology.orchestra.run.smithi027.stdout:mds dev mds_verify_scatter true 2024-09-21T03:26:55.494 INFO:teuthology.orchestra.run.smithi027.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-21T03:26:55.494 INFO:teuthology.orchestra.run.smithi027.stdout:mds advanced rados_mon_op_timeout 900 2024-09-21T03:26:55.494 INFO:teuthology.orchestra.run.smithi027.stdout:mds advanced rados_osd_op_timeout 900 2024-09-21T03:26:55.494 INFO:teuthology.orchestra.run.smithi027.stdout:client advanced client_mount_timeout 600 2024-09-21T03:26:55.494 INFO:teuthology.orchestra.run.smithi027.stdout:client advanced debug_client 20/20 2024-09-21T03:26:55.494 INFO:teuthology.orchestra.run.smithi027.stdout:client advanced debug_ms 1/1 2024-09-21T03:26:55.494 INFO:teuthology.orchestra.run.smithi027.stdout:client advanced rados_mon_op_timeout 900 2024-09-21T03:26:55.494 INFO:teuthology.orchestra.run.smithi027.stdout:client advanced rados_osd_op_timeout 900 2024-09-21T03:26:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.492+0000 7fbc52867640 1 -- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbc4c1329e0 msgr2=0x7fbc4c122370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.492+0000 7fbc52867640 1 --2- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbc4c1329e0 0x7fbc4c122370 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbc44005fd0 tx=0x7fbc4400ee60 comp rx=0 tx=0).stop 2024-09-21T03:26:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.492+0000 7fbc52867640 1 -- 172.21.15.27:0/936539666 shutdown_connections 2024-09-21T03:26:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.492+0000 7fbc52867640 1 --2- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbc4c1329e0 0x7fbc4c122370 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.492+0000 7fbc52867640 1 --2- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc4c096ad0 0x7fbc4c121e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.492+0000 7fbc52867640 1 --2- 172.21.15.27:0/936539666 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbc4c06e920 0x7fbc4c1218f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.492+0000 7fbc52867640 1 -- 172.21.15.27:0/936539666 >> 172.21.15.27:0/936539666 conn(0x7fbc4c099510 msgr2=0x7fbc4c08f370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.492+0000 7fbc52867640 1 -- 172.21.15.27:0/936539666 shutdown_connections 2024-09-21T03:26:55.495 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.492+0000 7fbc52867640 1 -- 172.21.15.27:0/936539666 wait complete. 2024-09-21T03:26:55.504 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T03:26:55.504 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd crush tunables default 2024-09-21T03:26:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.645+0000 7ff19c583640 1 -- 172.21.15.27:0/3911064984 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff1941333d0 msgr2=0x7ff1941357c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.645+0000 7ff19c583640 1 --2- 172.21.15.27:0/3911064984 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff1941333d0 0x7ff1941357c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009cd0 tx=0x7ff18c02a980 comp rx=0 tx=0).stop 2024-09-21T03:26:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.646+0000 7ff19c583640 1 -- 172.21.15.27:0/3911064984 shutdown_connections 2024-09-21T03:26:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.646+0000 7ff19c583640 1 --2- 172.21.15.27:0/3911064984 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff1941333d0 0x7ff1941357c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.646+0000 7ff19c583640 1 --2- 172.21.15.27:0/3911064984 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff19408db50 0x7ff194132e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.648 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.646+0000 7ff19c583640 1 -- 172.21.15.27:0/3911064984 >> 172.21.15.27:0/3911064984 conn(0x7ff1940a05b0 msgr2=0x7ff1940a09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.646+0000 7ff19c583640 1 -- 172.21.15.27:0/3911064984 shutdown_connections 2024-09-21T03:26:55.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.646+0000 7ff19c583640 1 -- 172.21.15.27:0/3911064984 wait complete. 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19c583640 1 Processor -- start 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19c583640 1 -- start start 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19c583640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff19408db50 0x7ff1941c3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19c583640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19412dfa0 0x7ff1941c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19c583640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff1941333d0 0x7ff194130fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19c583640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff194136970 con 0x7ff19412dfa0 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19c583640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7ff1941367f0 con 0x7ff1941333d0 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19c583640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7ff194136af0 con 0x7ff19408db50 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19ab20640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff1941333d0 0x7ff194130fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19a31f640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff19408db50 0x7ff1941c3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff199b1e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19412dfa0 0x7ff1941c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19a31f640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff19408db50 0x7ff1941c3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:55668/0 (socket says 172.21.15.27:55668) 2024-09-21T03:26:55.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19ab20640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff1941333d0 0x7ff194130fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:41362/0 (socket says 172.21.15.27:41362) 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.647+0000 7ff19a31f640 1 -- 172.21.15.27:0/202328418 learned_addr learned my addr 172.21.15.27:0/202328418 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff19ab20640 1 -- 172.21.15.27:0/202328418 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff19408db50 msgr2=0x7ff1941c3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff19ab20640 1 --2- 172.21.15.27:0/202328418 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff19408db50 0x7ff1941c3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff19ab20640 1 -- 172.21.15.27:0/202328418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19412dfa0 msgr2=0x7ff1941c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff19ab20640 1 --2- 172.21.15.27:0/202328418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19412dfa0 0x7ff1941c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff19ab20640 1 -- 172.21.15.27:0/202328418 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff194132b40 con 0x7ff1941333d0 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff19ab20640 1 --2- 172.21.15.27:0/202328418 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff1941333d0 0x7ff194130fd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff18c032000 tx=0x7ff18c031da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff18affd640 1 -- 172.21.15.27:0/202328418 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c034040 con 0x7ff1941333d0 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff18affd640 1 -- 172.21.15.27:0/202328418 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff18c004240 con 0x7ff1941333d0 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff18affd640 1 -- 172.21.15.27:0/202328418 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c004650 con 0x7ff1941333d0 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff19c583640 1 -- 172.21.15.27:0/202328418 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1941256e0 con 0x7ff1941333d0 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff19c583640 1 -- 172.21.15.27:0/202328418 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1941c4b20 con 0x7ff1941333d0 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.648+0000 7ff18affd640 1 -- 172.21.15.27:0/202328418 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff18c037020 con 0x7ff1941333d0 2024-09-21T03:26:55.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.649+0000 7ff18affd640 1 -- 172.21.15.27:0/202328418 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff18c039430 con 0x7ff1941333d0 2024-09-21T03:26:55.653 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.650+0000 7ff19c583640 1 -- 172.21.15.27:0/202328418 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1940a0a40 con 0x7ff1941333d0 2024-09-21T03:26:55.654 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.652+0000 7ff18affd640 1 -- 172.21.15.27:0/202328418 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7ff1940a0a40 con 0x7ff1941333d0 2024-09-21T03:26:55.688 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:55.686+0000 7f3c11c12040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T03:26:55.689 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:55.686+0000 7f2af1337040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T03:26:55.718 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.716+0000 7ff19c583640 1 -- 172.21.15.27:0/202328418 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7ff19412e380 con 0x7ff1941333d0 2024-09-21T03:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.721+0000 7ff18affd640 1 -- 172.21.15.27:0/202328418 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7ff18c004b40 con 0x7ff1941333d0 2024-09-21T03:26:55.723 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2024-09-21T03:26:55.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.722+0000 7ff19c583640 1 -- 172.21.15.27:0/202328418 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff1941333d0 msgr2=0x7ff194130fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.722+0000 7ff19c583640 1 --2- 172.21.15.27:0/202328418 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff1941333d0 0x7ff194130fd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff18c032000 tx=0x7ff18c031da0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.722+0000 7ff19c583640 1 -- 172.21.15.27:0/202328418 shutdown_connections 2024-09-21T03:26:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.722+0000 7ff19c583640 1 --2- 172.21.15.27:0/202328418 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff1941333d0 0x7ff194130fd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.722+0000 7ff19c583640 1 --2- 172.21.15.27:0/202328418 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff19412dfa0 0x7ff1941c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.722+0000 7ff19c583640 1 --2- 172.21.15.27:0/202328418 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff19408db50 0x7ff1941c3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.722+0000 7ff19c583640 1 -- 172.21.15.27:0/202328418 >> 172.21.15.27:0/202328418 conn(0x7ff1940a05b0 msgr2=0x7ff19406d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.722+0000 7ff19c583640 1 -- 172.21.15.27:0/202328418 shutdown_connections 2024-09-21T03:26:55.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:55.722+0000 7ff19c583640 1 -- 172.21.15.27:0/202328418 wait complete. 2024-09-21T03:26:55.735 INFO:tasks.ceph:check_enable_crimson: False 2024-09-21T03:26:55.735 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-21T03:26:55.735 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:55.736 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-21T03:26:55.746 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:55.744+0000 7f2af1337040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T03:26:55.746 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:55.744+0000 7f3c11c12040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T03:26:55.761 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:55.762 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-21T03:26:55.828 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:55.828 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-21T03:26:55.895 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:26:55.895 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-21T03:26:55.898 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:55.896+0000 7f3c11c12040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T03:26:55.898 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:55.896+0000 7f2af1337040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T03:26:55.923 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd new 5bfee00c-9ea5-41bd-b128-5eb75667f4f1 0 2024-09-21T03:26:56.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.067+0000 7f5d03838640 1 -- 172.21.15.27:0/157415735 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5cfc06ed60 msgr2=0x7f5cfc096010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.067+0000 7f5d03838640 1 --2- 172.21.15.27:0/157415735 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5cfc06ed60 0x7f5cfc096010 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5cec005fd0 tx=0x7f5cec02a4b0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.067+0000 7f5d03838640 1 -- 172.21.15.27:0/157415735 shutdown_connections 2024-09-21T03:26:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.067+0000 7f5d03838640 1 --2- 172.21.15.27:0/157415735 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5cfc06ed60 0x7f5cfc096010 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.067+0000 7f5d03838640 1 --2- 172.21.15.27:0/157415735 >> v2:172.21.15.27:3300/0 conn(0x7f5cfc06e440 0x7f5cfc06e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.067+0000 7f5d03838640 1 -- 172.21.15.27:0/157415735 >> 172.21.15.27:0/157415735 conn(0x7f5cfc09c930 msgr2=0x7f5cfc09cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.067+0000 7f5d03838640 1 -- 172.21.15.27:0/157415735 shutdown_connections 2024-09-21T03:26:56.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.067+0000 7f5d03838640 1 -- 172.21.15.27:0/157415735 wait complete. 2024-09-21T03:26:56.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.068+0000 7f5d03838640 1 Processor -- start 2024-09-21T03:26:56.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.068+0000 7f5d03838640 1 -- start start 2024-09-21T03:26:56.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.069+0000 7f5d03838640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc06e440 0x7f5cfc095270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.069+0000 7f5d03838640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5cfc06ed60 0x7f5cfc0957b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.069+0000 7f5d03838640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5cfc096550 0x7f5cfc125b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.069+0000 7f5d03838640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc134680 con 0x7f5cfc06e440 2024-09-21T03:26:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.069+0000 7f5d03838640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f5cfc096ea0 con 0x7f5cfc096550 2024-09-21T03:26:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.069+0000 7f5d015d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc06e440 0x7f5cfc095270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.069+0000 7f5d03838640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f5cfc134800 con 0x7f5cfc06ed60 2024-09-21T03:26:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.069+0000 7f5d015d4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc06e440 0x7f5cfc095270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:43148/0 (socket says 172.21.15.27:43148) 2024-09-21T03:26:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.069+0000 7f5d00dd3640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5cfc06ed60 0x7f5cfc0957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.069+0000 7f5d015d4640 1 -- 172.21.15.27:0/1352079413 learned_addr learned my addr 172.21.15.27:0/1352079413 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:56.074 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.069+0000 7f5d01dd5640 1 --2- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5cfc096550 0x7f5cfc125b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5d00dd3640 1 -- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5cfc096550 msgr2=0x7f5cfc125b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5d00dd3640 1 --2- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5cfc096550 0x7f5cfc125b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5d00dd3640 1 -- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc06e440 msgr2=0x7f5cfc095270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5d00dd3640 1 --2- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc06e440 0x7f5cfc095270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5d00dd3640 1 -- 172.21.15.27:0/1352079413 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cfc127620 con 0x7f5cfc06ed60 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5d01dd5640 1 --2- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5cfc096550 0x7f5cfc125b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5d00dd3640 1 --2- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5cfc06ed60 0x7f5cfc0957b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5cec031000 tx=0x7f5cec0038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5cf1ffb640 1 -- 172.21.15.27:0/1352079413 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cec03f070 con 0x7f5cfc06ed60 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5cf1ffb640 1 -- 172.21.15.27:0/1352079413 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5cec0047a0 con 0x7f5cfc06ed60 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5cf1ffb640 1 -- 172.21.15.27:0/1352079413 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cec035030 con 0x7f5cfc06ed60 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5d03838640 1 -- 172.21.15.27:0/1352079413 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cfc095cf0 con 0x7f5cfc06ed60 2024-09-21T03:26:56.075 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.070+0000 7f5d03838640 1 -- 172.21.15.27:0/1352079413 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cfc1288e0 con 0x7f5cfc06ed60 2024-09-21T03:26:56.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.072+0000 7f5cf1ffb640 1 -- 172.21.15.27:0/1352079413 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5cec042070 con 0x7f5cfc06ed60 2024-09-21T03:26:56.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.072+0000 7f5cf1ffb640 1 -- 172.21.15.27:0/1352079413 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f5cec03b9b0 con 0x7f5cfc06ed60 2024-09-21T03:26:56.076 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.072+0000 7f5d03838640 1 -- 172.21.15.27:0/1352079413 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cfc099dc0 con 0x7f5cfc06ed60 2024-09-21T03:26:56.078 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.076+0000 7f5cf1ffb640 1 -- 172.21.15.27:0/1352079413 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f5cfc099dc0 con 0x7f5cfc06ed60 2024-09-21T03:26:56.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.162+0000 7f5d03838640 1 -- 172.21.15.27:0/1352079413 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd new", "uuid": "5bfee00c-9ea5-41bd-b128-5eb75667f4f1", "id": 0} v 0) -- 0x7f5cfc06e820 con 0x7f5cfc06ed60 2024-09-21T03:26:56.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.167+0000 7f5cf1ffb640 1 -- 172.21.15.27:0/1352079413 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5bfee00c-9ea5-41bd-b128-5eb75667f4f1", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f5cec03bc00 con 0x7f5cfc06ed60 2024-09-21T03:26:56.169 INFO:teuthology.orchestra.run.smithi027.stdout:0 2024-09-21T03:26:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.168+0000 7f5d03838640 1 -- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5cfc06ed60 msgr2=0x7f5cfc0957b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.168+0000 7f5d03838640 1 --2- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5cfc06ed60 0x7f5cfc0957b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5cec031000 tx=0x7f5cec0038a0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.168+0000 7f5d03838640 1 -- 172.21.15.27:0/1352079413 shutdown_connections 2024-09-21T03:26:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.168+0000 7f5d03838640 1 --2- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5cfc096550 0x7f5cfc125b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.171 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.168+0000 7f5d03838640 1 --2- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5cfc06ed60 0x7f5cfc0957b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.168+0000 7f5d03838640 1 --2- 172.21.15.27:0/1352079413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cfc06e440 0x7f5cfc095270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.169+0000 7f5d03838640 1 -- 172.21.15.27:0/1352079413 >> 172.21.15.27:0/1352079413 conn(0x7f5cfc09c930 msgr2=0x7f5cfc0981c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:56.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.169+0000 7f5d03838640 1 -- 172.21.15.27:0/1352079413 shutdown_connections 2024-09-21T03:26:56.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.169+0000 7f5d03838640 1 -- 172.21.15.27:0/1352079413 wait complete. 2024-09-21T03:26:56.185 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd new 121ea55e-e6d7-4d7e-bbc8-ef3fac12060f 1 2024-09-21T03:26:56.279 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:56.276+0000 7f3c11c12040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T03:26:56.287 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:56.285+0000 7f2af1337040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T03:26:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.308+0000 7f6733118640 1 -- 172.21.15.27:0/2096726992 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f672c095dd0 msgr2=0x7f672c0961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.308+0000 7f6733118640 1 --2- 172.21.15.27:0/2096726992 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f672c095dd0 0x7f672c0961b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6720009830 tx=0x7f672002a350 comp rx=0 tx=0).stop 2024-09-21T03:26:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.308+0000 7f6733118640 1 -- 172.21.15.27:0/2096726992 shutdown_connections 2024-09-21T03:26:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.308+0000 7f6733118640 1 --2- 172.21.15.27:0/2096726992 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f672c1302b0 0x7f672c1326a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.308+0000 7f6733118640 1 --2- 172.21.15.27:0/2096726992 >> v2:172.21.15.27:3300/0 conn(0x7f672c0966f0 0x7f672c096af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.308+0000 7f6733118640 1 --2- 172.21.15.27:0/2096726992 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f672c095dd0 0x7f672c0961b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.308+0000 7f6733118640 1 -- 172.21.15.27:0/2096726992 >> 172.21.15.27:0/2096726992 conn(0x7f672c09e560 msgr2=0x7f672c09e970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:56.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.309+0000 7f6733118640 1 -- 172.21.15.27:0/2096726992 shutdown_connections 2024-09-21T03:26:56.312 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.309+0000 7f6733118640 1 -- 172.21.15.27:0/2096726992 wait complete. 2024-09-21T03:26:56.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.309+0000 7f6733118640 1 Processor -- start 2024-09-21T03:26:56.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.309+0000 7f6733118640 1 -- start start 2024-09-21T03:26:56.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f6733118640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f672c095dd0 0x7f672c12f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.314 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f6733118640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f672c0966f0 0x7f672c12fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f6733118640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f672c1302b0 0x7f672c08dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f6733118640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f672c133880 con 0x7f672c095dd0 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f6733118640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f672c133700 con 0x7f672c0966f0 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f6733118640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f672c133a00 con 0x7f672c1302b0 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f672bfff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f672c0966f0 0x7f672c12fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f672bfff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f672c0966f0 0x7f672c12fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:41382/0 (socket says 172.21.15.27:41382) 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f672bfff640 1 -- 172.21.15.27:0/71992920 learned_addr learned my addr 172.21.15.27:0/71992920 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f67316b5640 1 --2- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f672c1302b0 0x7f672c08dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f6730eb4640 1 --2- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f672c095dd0 0x7f672c12f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f672bfff640 1 -- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f672c1302b0 msgr2=0x7f672c08dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f672bfff640 1 --2- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f672c1302b0 0x7f672c08dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f672bfff640 1 -- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f672c095dd0 msgr2=0x7f672c12f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.310+0000 7f672bfff640 1 --2- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f672c095dd0 0x7f672c12f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.311+0000 7f672bfff640 1 -- 172.21.15.27:0/71992920 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672c08f730 con 0x7f672c0966f0 2024-09-21T03:26:56.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.311+0000 7f6730eb4640 1 --2- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f672c095dd0 0x7f672c12f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:26:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.311+0000 7f672bfff640 1 --2- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f672c0966f0 0x7f672c12fa10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f671800ea60 tx=0x7f671800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.311+0000 7f67297fa640 1 -- 172.21.15.27:0/71992920 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671800cde0 con 0x7f672c0966f0 2024-09-21T03:26:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.311+0000 7f67297fa640 1 -- 172.21.15.27:0/71992920 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6718004020 con 0x7f672c0966f0 2024-09-21T03:26:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.311+0000 7f6733118640 1 -- 172.21.15.27:0/71992920 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672c08fe20 con 0x7f672c0966f0 2024-09-21T03:26:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.311+0000 7f67297fa640 1 -- 172.21.15.27:0/71992920 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718004450 con 0x7f672c0966f0 2024-09-21T03:26:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.311+0000 7f6733118640 1 -- 172.21.15.27:0/71992920 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f672c090880 con 0x7f672c0966f0 2024-09-21T03:26:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.311+0000 7f67297fa640 1 -- 172.21.15.27:0/71992920 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f67180053b0 con 0x7f672c0966f0 2024-09-21T03:26:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.311+0000 7f67297fa640 1 -- 172.21.15.27:0/71992920 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f6718003c50 con 0x7f672c0966f0 2024-09-21T03:26:56.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.312+0000 7f6733118640 1 -- 172.21.15.27:0/71992920 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f672c0927e0 con 0x7f672c0966f0 2024-09-21T03:26:56.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.314+0000 7f67297fa640 1 -- 172.21.15.27:0/71992920 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f672c0927e0 con 0x7f672c0966f0 2024-09-21T03:26:56.337 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:56.335+0000 7f3c11c12040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T03:26:56.338 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:56.336+0000 7f2af1337040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T03:26:56.389 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.386+0000 7f6733118640 1 -- 172.21.15.27:0/71992920 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd new", "uuid": "121ea55e-e6d7-4d7e-bbc8-ef3fac12060f", "id": 1} v 0) -- 0x7f672c0961b0 con 0x7f672c0966f0 2024-09-21T03:26:56.392 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.390+0000 7f67297fa640 1 -- 172.21.15.27:0/71992920 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "121ea55e-e6d7-4d7e-bbc8-ef3fac12060f", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f67180047c0 con 0x7f672c0966f0 2024-09-21T03:26:56.392 INFO:teuthology.orchestra.run.smithi027.stdout:1 2024-09-21T03:26:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.392+0000 7f6733118640 1 -- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f672c0966f0 msgr2=0x7f672c12fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.392+0000 7f6733118640 1 --2- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f672c0966f0 0x7f672c12fa10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f671800ea60 tx=0x7f671800ef20 comp rx=0 tx=0).stop 2024-09-21T03:26:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.392+0000 7f6733118640 1 -- 172.21.15.27:0/71992920 shutdown_connections 2024-09-21T03:26:56.394 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.392+0000 7f6733118640 1 --2- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f672c1302b0 0x7f672c08dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.392+0000 7f6733118640 1 --2- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f672c0966f0 0x7f672c12fa10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.392+0000 7f6733118640 1 --2- 172.21.15.27:0/71992920 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f672c095dd0 0x7f672c12f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.392+0000 7f6733118640 1 -- 172.21.15.27:0/71992920 >> 172.21.15.27:0/71992920 conn(0x7f672c09e560 msgr2=0x7f672c06fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.392+0000 7f6733118640 1 -- 172.21.15.27:0/71992920 shutdown_connections 2024-09-21T03:26:56.395 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.392+0000 7f6733118640 1 -- 172.21.15.27:0/71992920 wait complete. 2024-09-21T03:26:56.409 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd new b0834a51-8a1a-4bdf-9e82-9b77d445c2a5 2 2024-09-21T03:26:56.429 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:56.427+0000 7f2af1337040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T03:26:56.434 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:56.432+0000 7f3c11c12040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T03:26:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.510+0000 7f419d4b1640 1 -- 172.21.15.27:0/620653915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4198091a00 msgr2=0x7f4198132810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.510+0000 7f419d4b1640 1 --2- 172.21.15.27:0/620653915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4198091a00 0x7f4198132810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4190005fd0 tx=0x7f419002a980 comp rx=0 tx=0).stop 2024-09-21T03:26:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.510+0000 7f419d4b1640 1 -- 172.21.15.27:0/620653915 shutdown_connections 2024-09-21T03:26:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.510+0000 7f419d4b1640 1 --2- 172.21.15.27:0/620653915 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4198091a00 0x7f4198132810 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.510+0000 7f419d4b1640 1 --2- 172.21.15.27:0/620653915 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f41980970b0 0x7f41980910c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.510+0000 7f419d4b1640 1 -- 172.21.15.27:0/620653915 >> 172.21.15.27:0/620653915 conn(0x7f419809d150 msgr2=0x7f419809d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.510+0000 7f419d4b1640 1 -- 172.21.15.27:0/620653915 shutdown_connections 2024-09-21T03:26:56.513 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.510+0000 7f419d4b1640 1 -- 172.21.15.27:0/620653915 wait complete. 2024-09-21T03:26:56.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.511+0000 7f419d4b1640 1 Processor -- start 2024-09-21T03:26:56.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.511+0000 7f419d4b1640 1 -- start start 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f419d4b1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4198091600 0x7f419808cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f419d4b1640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4198091a00 0x7f419808d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f419d4b1640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f41980970b0 0x7f419808d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f419d4b1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4198133690 con 0x7f4198091600 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f419d4b1640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f4198133510 con 0x7f4198091a00 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f419d4b1640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f4198133810 con 0x7f41980970b0 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f41967fc640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4198091a00 0x7f419808d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f41967fc640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4198091a00 0x7f419808d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:41398/0 (socket says 172.21.15.27:41398) 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f41967fc640 1 -- 172.21.15.27:0/1844968808 learned_addr learned my addr 172.21.15.27:0/1844968808 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f41977fe640 1 --2- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f41980970b0 0x7f419808d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f4196ffd640 1 --2- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4198091600 0x7f419808cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f41967fc640 1 -- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f41980970b0 msgr2=0x7f419808d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f41967fc640 1 --2- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f41980970b0 0x7f419808d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f41967fc640 1 -- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4198091600 msgr2=0x7f419808cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f41967fc640 1 --2- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4198091600 0x7f419808cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f41967fc640 1 -- 172.21.15.27:0/1844968808 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419808e4c0 con 0x7f4198091a00 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f4196ffd640 1 --2- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4198091600 0x7f419808cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.512+0000 7f41967fc640 1 --2- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4198091a00 0x7f419808d140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f418c00c450 tx=0x7f418c00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.513+0000 7f417f7fe640 1 -- 172.21.15.27:0/1844968808 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c0052d0 con 0x7f4198091a00 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.513+0000 7f417f7fe640 1 -- 172.21.15.27:0/1844968808 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f418c005470 con 0x7f4198091a00 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.513+0000 7f417f7fe640 1 -- 172.21.15.27:0/1844968808 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c012870 con 0x7f4198091a00 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.513+0000 7f419d4b1640 1 -- 172.21.15.27:0/1844968808 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419808dbc0 con 0x7f4198091a00 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.513+0000 7f419d4b1640 1 -- 172.21.15.27:0/1844968808 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f419808f780 con 0x7f4198091a00 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.513+0000 7f417f7fe640 1 -- 172.21.15.27:0/1844968808 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f418c012a10 con 0x7f4198091a00 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.513+0000 7f417f7fe640 1 -- 172.21.15.27:0/1844968808 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f418c012d00 con 0x7f4198091a00 2024-09-21T03:26:56.517 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.514+0000 7f419d4b1640 1 -- 172.21.15.27:0/1844968808 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41980910c0 con 0x7f4198091a00 2024-09-21T03:26:56.520 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.518+0000 7f417f7fe640 1 -- 172.21.15.27:0/1844968808 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f418c0035f0 con 0x7f4198091a00 2024-09-21T03:26:56.537 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:56.535+0000 7f3c11c12040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T03:26:56.543 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:56.541+0000 7f2af1337040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T03:26:56.587 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:56.584+0000 7f3c11c12040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T03:26:56.593 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:56.590+0000 7f2af1337040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T03:26:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.608+0000 7f419d4b1640 1 -- 172.21.15.27:0/1844968808 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b0834a51-8a1a-4bdf-9e82-9b77d445c2a5", "id": 2} v 0) -- 0x7f41980689e0 con 0x7f4198091a00 2024-09-21T03:26:56.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.611+0000 7f417f7fe640 1 -- 172.21.15.27:0/1844968808 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b0834a51-8a1a-4bdf-9e82-9b77d445c2a5", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f418c003970 con 0x7f4198091a00 2024-09-21T03:26:56.614 INFO:teuthology.orchestra.run.smithi027.stdout:2 2024-09-21T03:26:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.612+0000 7f419d4b1640 1 -- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4198091a00 msgr2=0x7f419808d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.612+0000 7f419d4b1640 1 --2- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4198091a00 0x7f419808d140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f418c00c450 tx=0x7f418c00c910 comp rx=0 tx=0).stop 2024-09-21T03:26:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.612+0000 7f419d4b1640 1 -- 172.21.15.27:0/1844968808 shutdown_connections 2024-09-21T03:26:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.612+0000 7f419d4b1640 1 --2- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f41980970b0 0x7f419808d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.612+0000 7f419d4b1640 1 --2- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f4198091a00 0x7f419808d140 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.612+0000 7f419d4b1640 1 --2- 172.21.15.27:0/1844968808 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4198091600 0x7f419808cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.612+0000 7f419d4b1640 1 -- 172.21.15.27:0/1844968808 >> 172.21.15.27:0/1844968808 conn(0x7f419809d150 msgr2=0x7f419806bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.612+0000 7f419d4b1640 1 -- 172.21.15.27:0/1844968808 shutdown_connections 2024-09-21T03:26:56.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.612+0000 7f419d4b1640 1 -- 172.21.15.27:0/1844968808 wait complete. 2024-09-21T03:26:56.625 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd new 344f8d01-27ba-4598-a138-5fdde9141b60 3 2024-09-21T03:26:56.696 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:56.694+0000 7f3c11c12040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T03:26:56.698 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:56.696+0000 7f2af1337040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T03:26:56.754 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:56.751+0000 7f3c11c12040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T03:26:56.852 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:56.751+0000 7f2af1337040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T03:26:56.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.768+0000 7f2d95133640 1 -- 172.21.15.27:0/3314887533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d901333d0 msgr2=0x7f2d901357c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.768+0000 7f2d95133640 1 --2- 172.21.15.27:0/3314887533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d901333d0 0x7f2d901357c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2d80005fd0 tx=0x7f2d8002a980 comp rx=0 tx=0).stop 2024-09-21T03:26:56.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.768+0000 7f2d95133640 1 -- 172.21.15.27:0/3314887533 shutdown_connections 2024-09-21T03:26:56.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.768+0000 7f2d95133640 1 --2- 172.21.15.27:0/3314887533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d901333d0 0x7f2d901357c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.768+0000 7f2d95133640 1 --2- 172.21.15.27:0/3314887533 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2d9008db50 0x7f2d90132e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.768+0000 7f2d95133640 1 -- 172.21.15.27:0/3314887533 >> 172.21.15.27:0/3314887533 conn(0x7f2d900a05b0 msgr2=0x7f2d900a09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.768+0000 7f2d95133640 1 -- 172.21.15.27:0/3314887533 shutdown_connections 2024-09-21T03:26:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.768+0000 7f2d95133640 1 -- 172.21.15.27:0/3314887533 wait complete. 2024-09-21T03:26:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.769+0000 7f2d95133640 1 Processor -- start 2024-09-21T03:26:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.769+0000 7f2d95133640 1 -- start start 2024-09-21T03:26:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.769+0000 7f2d95133640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2d9008db50 0x7f2d90130ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.769+0000 7f2d95133640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2d9012dfa0 0x7f2d90131530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d95133640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d901333d0 0x7f2d90131a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d95133640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2d90136910 con 0x7f2d901333d0 2024-09-21T03:26:56.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d95133640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f2d90136790 con 0x7f2d9012dfa0 2024-09-21T03:26:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d95133640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f2d90136a90 con 0x7f2d9008db50 2024-09-21T03:26:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8ed9d640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2d9008db50 0x7f2d90130ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d901333d0 0x7f2d90131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8ed9d640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2d9008db50 0x7f2d90130ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:55716/0 (socket says 172.21.15.27:55716) 2024-09-21T03:26:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8ed9d640 1 -- 172.21.15.27:0/168972627 learned_addr learned my addr 172.21.15.27:0/168972627 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8e59c640 1 --2- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2d9012dfa0 0x7f2d90131530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8ed9d640 1 -- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2d9012dfa0 msgr2=0x7f2d90131530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8ed9d640 1 --2- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2d9012dfa0 0x7f2d90131530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8ed9d640 1 -- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d901333d0 msgr2=0x7f2d90131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8ed9d640 1 --2- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d901333d0 0x7f2d90131a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8ed9d640 1 -- 172.21.15.27:0/168972627 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d90092d40 con 0x7f2d9008db50 2024-09-21T03:26:56.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8f59e640 1 --2- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d901333d0 0x7f2d90131a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:26:56.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.770+0000 7f2d8ed9d640 1 --2- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2d9008db50 0x7f2d90130ff0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2d8800c450 tx=0x7f2d8800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:56.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.771+0000 7f2d877fe640 1 -- 172.21.15.27:0/168972627 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d880052d0 con 0x7f2d9008db50 2024-09-21T03:26:56.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.771+0000 7f2d877fe640 1 -- 172.21.15.27:0/168972627 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2d88005470 con 0x7f2d9008db50 2024-09-21T03:26:56.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.771+0000 7f2d877fe640 1 -- 172.21.15.27:0/168972627 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d88012870 con 0x7f2d9008db50 2024-09-21T03:26:56.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.771+0000 7f2d95133640 1 -- 172.21.15.27:0/168972627 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d90126820 con 0x7f2d9008db50 2024-09-21T03:26:56.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.771+0000 7f2d95133640 1 -- 172.21.15.27:0/168972627 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d90131fb0 con 0x7f2d9008db50 2024-09-21T03:26:56.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.771+0000 7f2d877fe640 1 -- 172.21.15.27:0/168972627 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2d88012a10 con 0x7f2d9008db50 2024-09-21T03:26:56.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.771+0000 7f2d877fe640 1 -- 172.21.15.27:0/168972627 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f2d8801b930 con 0x7f2d9008db50 2024-09-21T03:26:56.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.772+0000 7f2d95133640 1 -- 172.21.15.27:0/168972627 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d9012e380 con 0x7f2d9008db50 2024-09-21T03:26:56.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.774+0000 7f2d877fe640 1 -- 172.21.15.27:0/168972627 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f2d90092d40 con 0x7f2d9008db50 2024-09-21T03:26:56.859 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:56.800+0000 7f2af1337040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T03:26:56.859 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:56.811+0000 7f3c11c12040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T03:26:56.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.873+0000 7f2d95133640 1 -- 172.21.15.27:0/168972627 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd new", "uuid": "344f8d01-27ba-4598-a138-5fdde9141b60", "id": 3} v 0) -- 0x7f2d900689e0 con 0x7f2d9008db50 2024-09-21T03:26:56.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.877+0000 7f2d877fe640 1 -- 172.21.15.27:0/168972627 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "344f8d01-27ba-4598-a138-5fdde9141b60", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f2d8801b5b0 con 0x7f2d9008db50 2024-09-21T03:26:56.880 INFO:teuthology.orchestra.run.smithi027.stdout:3 2024-09-21T03:26:56.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.879+0000 7f2d95133640 1 -- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2d9008db50 msgr2=0x7f2d90130ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:56.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.879+0000 7f2d95133640 1 --2- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2d9008db50 0x7f2d90130ff0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2d8800c450 tx=0x7f2d8800c910 comp rx=0 tx=0).stop 2024-09-21T03:26:56.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.879+0000 7f2d95133640 1 -- 172.21.15.27:0/168972627 shutdown_connections 2024-09-21T03:26:56.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.879+0000 7f2d95133640 1 --2- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d901333d0 0x7f2d90131a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.881 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.879+0000 7f2d95133640 1 --2- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2d9012dfa0 0x7f2d90131530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.879+0000 7f2d95133640 1 --2- 172.21.15.27:0/168972627 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2d9008db50 0x7f2d90130ff0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:56.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.879+0000 7f2d95133640 1 -- 172.21.15.27:0/168972627 >> 172.21.15.27:0/168972627 conn(0x7f2d900a05b0 msgr2=0x7f2d9012c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:56.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.879+0000 7f2d95133640 1 -- 172.21.15.27:0/168972627 shutdown_connections 2024-09-21T03:26:56.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:56.879+0000 7f2d95133640 1 -- 172.21.15.27:0/168972627 wait complete. 2024-09-21T03:26:56.895 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T03:26:56.895 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-21T03:26:56.920 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:56.918+0000 7f2af1337040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T03:26:56.922 INFO:tasks.ceph.osd.0:Started 2024-09-21T03:26:56.922 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-21T03:26:56.922 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-21T03:26:56.925 INFO:tasks.ceph.osd.1:Started 2024-09-21T03:26:56.925 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T03:26:56.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-21T03:26:56.929 INFO:tasks.ceph.osd.2:Started 2024-09-21T03:26:56.929 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-21T03:26:56.929 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-21T03:26:56.931 INFO:tasks.ceph.osd.3:Started 2024-09-21T03:26:56.932 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:26:56.935 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:56.932+0000 7f3c11c12040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T03:26:57.003 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:57.001+0000 7f2af1337040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T03:26:57.018 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:57.016+0000 7f3c11c12040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T03:26:57.085 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:57.083+0000 7f3c11c12040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T03:26:57.087 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:57.085+0000 7f2af1337040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T03:26:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.089+0000 7fb43979d640 1 -- 172.21.15.27:0/3713545703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb43406ed90 msgr2=0x7fb434130e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.089+0000 7fb43979d640 1 --2- 172.21.15.27:0/3713545703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb43406ed90 0x7fb434130e50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb428005fd0 tx=0x7fb42802a4b0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.089+0000 7fb43979d640 1 -- 172.21.15.27:0/3713545703 shutdown_connections 2024-09-21T03:26:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.089+0000 7fb43979d640 1 --2- 172.21.15.27:0/3713545703 >> v2:172.21.15.27:3300/0 conn(0x7fb434131390 0x7fb434133780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.089+0000 7fb43979d640 1 --2- 172.21.15.27:0/3713545703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb43406ed90 0x7fb434130e50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.089+0000 7fb43979d640 1 -- 172.21.15.27:0/3713545703 >> 172.21.15.27:0/3713545703 conn(0x7fb43409d300 msgr2=0x7fb43409d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.089+0000 7fb43979d640 1 -- 172.21.15.27:0/3713545703 shutdown_connections 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.089+0000 7fb43979d640 1 -- 172.21.15.27:0/3713545703 wait complete. 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.090+0000 7fb43979d640 1 Processor -- start 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.090+0000 7fb43979d640 1 -- start start 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb43979d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb43406d840 0x7fb43412ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb43979d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43406ed90 0x7fb43412d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb43979d640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb434131390 0x7fb43412d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb43979d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb4340953e0 con 0x7fb43406ed90 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb43979d640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fb434095260 con 0x7fb43406d840 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb43979d640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fb434095560 con 0x7fb434131390 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb438f9c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb434131390 0x7fb43412d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:57.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb4337fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43406ed90 0x7fb43412d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:57.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb433fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb43406d840 0x7fb43412ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:57.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb438f9c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb434131390 0x7fb43412d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:55760/0 (socket says 172.21.15.27:55760) 2024-09-21T03:26:57.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb438f9c640 1 -- 172.21.15.27:0/2339978995 learned_addr learned my addr 172.21.15.27:0/2339978995 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:57.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb438f9c640 1 -- 172.21.15.27:0/2339978995 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb43406d840 msgr2=0x7fb43412ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:57.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb438f9c640 1 --2- 172.21.15.27:0/2339978995 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb43406d840 0x7fb43412ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb438f9c640 1 -- 172.21.15.27:0/2339978995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43406ed90 msgr2=0x7fb43412d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:57.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb438f9c640 1 --2- 172.21.15.27:0/2339978995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43406ed90 0x7fb43412d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.091+0000 7fb438f9c640 1 -- 172.21.15.27:0/2339978995 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4341306e0 con 0x7fb434131390 2024-09-21T03:26:57.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.092+0000 7fb438f9c640 1 --2- 172.21.15.27:0/2339978995 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb434131390 0x7fb43412d8a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009f60 tx=0x7fb42c009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:57.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.092+0000 7fb430ff9640 1 -- 172.21.15.27:0/2339978995 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c01c030 con 0x7fb434131390 2024-09-21T03:26:57.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.092+0000 7fb430ff9640 1 -- 172.21.15.27:0/2339978995 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb42c003970 con 0x7fb434131390 2024-09-21T03:26:57.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.092+0000 7fb43979d640 1 -- 172.21.15.27:0/2339978995 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43412f590 con 0x7fb434131390 2024-09-21T03:26:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.092+0000 7fb430ff9640 1 -- 172.21.15.27:0/2339978995 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb42c003d40 con 0x7fb434131390 2024-09-21T03:26:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.092+0000 7fb43979d640 1 -- 172.21.15.27:0/2339978995 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fb434130050 con 0x7fb434131390 2024-09-21T03:26:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.093+0000 7fb430ff9640 1 -- 172.21.15.27:0/2339978995 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb42c01f070 con 0x7fb434131390 2024-09-21T03:26:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.093+0000 7fb430ff9640 1 -- 172.21.15.27:0/2339978995 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fb42c0047c0 con 0x7fb434131390 2024-09-21T03:26:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.093+0000 7fb43979d640 1 -- 172.21.15.27:0/2339978995 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb43409cc90 con 0x7fb434131390 2024-09-21T03:26:57.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.095+0000 7fb430ff9640 1 -- 172.21.15.27:0/2339978995 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fb43409cc90 con 0x7fb434131390 2024-09-21T03:26:57.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.157+0000 7fb43979d640 1 -- 172.21.15.27:0/2339978995 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb434067a10 con 0x7fb434131390 2024-09-21T03:26:57.159 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:57.157+0000 7f3c11c12040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T03:26:57.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.157+0000 7fb430ff9640 1 -- 172.21.15.27:0/2339978995 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) ==== 74+0+4153 (secure 0 0 0) 0x7fb42c026070 con 0x7fb434131390 2024-09-21T03:26:57.160 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:26:57.160 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":6,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:26:56.875414+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:26:57.160 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:57.157+0000 7f2af1337040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T03:26:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.158+0000 7fb43979d640 1 -- 172.21.15.27:0/2339978995 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb434131390 msgr2=0x7fb43412d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.158+0000 7fb43979d640 1 --2- 172.21.15.27:0/2339978995 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb434131390 0x7fb43412d8a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb42c009f60 tx=0x7fb42c009f90 comp rx=0 tx=0).stop 2024-09-21T03:26:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.158+0000 7fb43979d640 1 -- 172.21.15.27:0/2339978995 shutdown_connections 2024-09-21T03:26:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.158+0000 7fb43979d640 1 --2- 172.21.15.27:0/2339978995 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb434131390 0x7fb43412d8a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.158+0000 7fb43979d640 1 --2- 172.21.15.27:0/2339978995 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb43406ed90 0x7fb43412d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.158+0000 7fb43979d640 1 --2- 172.21.15.27:0/2339978995 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb43406d840 0x7fb43412ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.161 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.158+0000 7fb43979d640 1 -- 172.21.15.27:0/2339978995 >> 172.21.15.27:0/2339978995 conn(0x7fb43409d300 msgr2=0x7fb434128480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:57.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.158+0000 7fb43979d640 1 -- 172.21.15.27:0/2339978995 shutdown_connections 2024-09-21T03:26:57.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:26:57.158+0000 7fb43979d640 1 -- 172.21.15.27:0/2339978995 wait complete. 2024-09-21T03:26:57.171 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-21T03:26:57.171 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T03:26:57.207 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:57.205+0000 7f3c11c12040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T03:26:57.208 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:57.205+0000 7f2af1337040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T03:26:57.255 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:57.253+0000 7f2af1337040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T03:26:57.256 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:57.253+0000 7f3c11c12040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T03:26:57.494 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:57.492+0000 7f2af1337040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T03:26:57.494 INFO:tasks.ceph.mgr.x.smithi027.stderr:2024-09-21T03:26:57.492+0000 7f3c11c12040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T03:26:57.574 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T03:26:57.669 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.666+0000 7f9fa0238640 1 -- 172.21.15.27:0/4097095011 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9f9812b040 msgr2=0x7f9f9812db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:57.669 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.666+0000 7f9fa0238640 1 --2- 172.21.15.27:0/4097095011 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9f9812b040 0x7f9f9812db50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009830 tx=0x7f9f8802a360 comp rx=0 tx=0).stop 2024-09-21T03:26:57.670 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.667+0000 7f9fa0238640 1 -- 172.21.15.27:0/4097095011 shutdown_connections 2024-09-21T03:26:57.670 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.667+0000 7f9fa0238640 1 --2- 172.21.15.27:0/4097095011 >> v2:172.21.15.27:3300/0 conn(0x7f9f9812e090 0x7f9f9812e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.670 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.667+0000 7f9fa0238640 1 --2- 172.21.15.27:0/4097095011 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9f9812b040 0x7f9f9812db50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.670 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.667+0000 7f9fa0238640 1 --2- 172.21.15.27:0/4097095011 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9f9812a5f0 0x7f9f9812ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.670 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.667+0000 7f9fa0238640 1 -- 172.21.15.27:0/4097095011 >> 172.21.15.27:0/4097095011 conn(0x7f9f980a0690 msgr2=0x7f9f980a0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:57.670 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.667+0000 7f9fa0238640 1 -- 172.21.15.27:0/4097095011 shutdown_connections 2024-09-21T03:26:57.670 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.667+0000 7f9fa0238640 1 -- 172.21.15.27:0/4097095011 wait complete. 2024-09-21T03:26:57.672 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.667+0000 7f9fa0238640 1 Processor -- start 2024-09-21T03:26:57.672 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9fa0238640 1 -- start start 2024-09-21T03:26:57.672 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9fa0238640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9f9812a5f0 0x7f9f98123780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:57.672 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9fa0238640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9f9812b040 0x7f9f98123cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:57.672 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9fa0238640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9812e090 0x7f9f98131b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:26:57.672 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9fa0238640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9f98135910 con 0x7f9f9812e090 2024-09-21T03:26:57.672 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9fa0238640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f9f98135790 con 0x7f9f9812a5f0 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9fa0238640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f9f98135a90 con 0x7f9f9812b040 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9dfd4640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9f9812a5f0 0x7f9f98123780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9e7d5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9812e090 0x7f9f98131b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9d7d3640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9f9812b040 0x7f9f98123cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9dfd4640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9f9812a5f0 0x7f9f98123780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:41562/0 (socket says 172.21.15.27:41562) 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9dfd4640 1 -- 172.21.15.27:0/3924202866 learned_addr learned my addr 172.21.15.27:0/3924202866 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9d7d3640 1 -- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9f9812a5f0 msgr2=0x7f9f98123780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9d7d3640 1 --2- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9f9812a5f0 0x7f9f98123780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9d7d3640 1 -- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9812e090 msgr2=0x7f9f98131b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9d7d3640 1 --2- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9812e090 0x7f9f98131b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9d7d3640 1 -- 172.21.15.27:0/3924202866 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f98124200 con 0x7f9f9812b040 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9dfd4640 1 --2- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9f9812a5f0 0x7f9f98123780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:26:57.673 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.668+0000 7f9f9e7d5640 1 --2- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9812e090 0x7f9f98131b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:26:57.674 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.669+0000 7f9f9d7d3640 1 --2- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9f9812b040 0x7f9f98123cc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9f98072130 tx=0x7f9f88033b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:26:57.674 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.669+0000 7f9f8e7fc640 1 -- 172.21.15.27:0/3924202866 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f8803f020 con 0x7f9f9812b040 2024-09-21T03:26:57.674 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.669+0000 7f9f8e7fc640 1 -- 172.21.15.27:0/3924202866 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9f88033d60 con 0x7f9f9812b040 2024-09-21T03:26:57.674 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.669+0000 7f9fa0238640 1 -- 172.21.15.27:0/3924202866 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f98133eb0 con 0x7f9f9812b040 2024-09-21T03:26:57.674 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.669+0000 7f9f8e7fc640 1 -- 172.21.15.27:0/3924202866 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f88037040 con 0x7f9f9812b040 2024-09-21T03:26:57.674 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.669+0000 7f9fa0238640 1 -- 172.21.15.27:0/3924202866 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f981320c0 con 0x7f9f9812b040 2024-09-21T03:26:57.674 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.670+0000 7f9f8e7fc640 1 -- 172.21.15.27:0/3924202866 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 2) ==== 28070+0+0 (secure 0 0 0) 0x7f9f88042020 con 0x7f9f9812b040 2024-09-21T03:26:57.674 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.670+0000 7f9f8e7fc640 1 -- 172.21.15.27:0/3924202866 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f9f88054010 con 0x7f9f9812b040 2024-09-21T03:26:57.674 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.670+0000 7f9fa0238640 1 -- 172.21.15.27:0/3924202866 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f981336b0 con 0x7f9f9812b040 2024-09-21T03:26:57.674 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.672+0000 7f9f8e7fc640 1 -- 172.21.15.27:0/3924202866 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f9f981336b0 con 0x7f9f9812b040 2024-09-21T03:26:57.733 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.731+0000 7f9fa0238640 1 -- 172.21.15.27:0/3924202866 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f9812db50 con 0x7f9f9812b040 2024-09-21T03:26:57.734 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.731+0000 7f9f8e7fc640 1 -- 172.21.15.27:0/3924202866 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v6) ==== 74+0+4153 (secure 0 0 0) 0x7f9f88049030 con 0x7f9f9812b040 2024-09-21T03:26:57.734 INFO:teuthology.misc.health.smithi027.stdout: 2024-09-21T03:26:57.734 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":6,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:26:56.875414+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:26:57.735 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.732+0000 7f9fa0238640 1 -- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9f9812b040 msgr2=0x7f9f98123cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:26:57.735 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.732+0000 7f9fa0238640 1 --2- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9f9812b040 0x7f9f98123cc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9f98072130 tx=0x7f9f88033b70 comp rx=0 tx=0).stop 2024-09-21T03:26:57.735 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.732+0000 7f9fa0238640 1 -- 172.21.15.27:0/3924202866 shutdown_connections 2024-09-21T03:26:57.735 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.732+0000 7f9fa0238640 1 --2- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f9812e090 0x7f9f98131b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.736 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.732+0000 7f9fa0238640 1 --2- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9f9812b040 0x7f9f98123cc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.736 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.732+0000 7f9fa0238640 1 --2- 172.21.15.27:0/3924202866 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9f9812a5f0 0x7f9f98123780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:26:57.736 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.732+0000 7f9fa0238640 1 -- 172.21.15.27:0/3924202866 >> 172.21.15.27:0/3924202866 conn(0x7f9f980a0690 msgr2=0x7f9f9806d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:26:57.736 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.732+0000 7f9fa0238640 1 -- 172.21.15.27:0/3924202866 shutdown_connections 2024-09-21T03:26:57.736 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:26:57.732+0000 7f9fa0238640 1 -- 172.21.15.27:0/3924202866 wait complete. 2024-09-21T03:26:57.743 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-21T03:26:58.507 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:58.505+0000 7f2ada101640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T03:26:58.507 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:26:58.505+0000 7f2ada101640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-21T03:26:58.670 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-09-21T03:26:58.667+0000 7fb8d2e18640 -1 Falling back to public interface 2024-09-21T03:26:58.670 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-09-21T03:26:58.668+0000 7efd1c40b640 -1 Falling back to public interface 2024-09-21T03:26:58.858 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-21T03:26:58.856+0000 7f3a10f20640 -1 Falling back to public interface 2024-09-21T03:26:58.898 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-09-21T03:26:58.895+0000 7fbf4f8fe640 -1 Falling back to public interface 2024-09-21T03:27:02.471 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-09-21T03:27:02.469+0000 7efd1c40b640 -1 osd.1 0 log_to_monitors true 2024-09-21T03:27:02.612 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-21T03:27:02.610+0000 7f3a10f20640 -1 osd.0 0 log_to_monitors true 2024-09-21T03:27:02.933 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-09-21T03:27:02.931+0000 7fb8d2e18640 -1 osd.2 0 log_to_monitors true 2024-09-21T03:27:03.165 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-09-21T03:27:03.163+0000 7fbf4f8fe640 -1 osd.3 0 log_to_monitors true 2024-09-21T03:27:04.146 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T03:27:04.254 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.251+0000 7f6bf98fc640 1 -- 172.21.15.27:0/2067674080 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bf4071150 msgr2=0x7f6bf4121dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:04.254 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.251+0000 7f6bf98fc640 1 --2- 172.21.15.27:0/2067674080 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bf4071150 0x7f6bf4121dd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009830 tx=0x7f6be802a360 comp rx=0 tx=0).stop 2024-09-21T03:27:04.254 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.251+0000 7f6bf98fc640 1 -- 172.21.15.27:0/2067674080 shutdown_connections 2024-09-21T03:27:04.254 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.251+0000 7f6bf98fc640 1 --2- 172.21.15.27:0/2067674080 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bf4122310 0x7f6bf4122760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:04.254 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.251+0000 7f6bf98fc640 1 --2- 172.21.15.27:0/2067674080 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bf4071150 0x7f6bf4121dd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:04.254 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.251+0000 7f6bf98fc640 1 --2- 172.21.15.27:0/2067674080 >> v2:172.21.15.27:3300/0 conn(0x7f6bf4070830 0x7f6bf4070c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:04.254 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.251+0000 7f6bf98fc640 1 -- 172.21.15.27:0/2067674080 >> 172.21.15.27:0/2067674080 conn(0x7f6bf409c420 msgr2=0x7f6bf409c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:04.254 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.251+0000 7f6bf98fc640 1 -- 172.21.15.27:0/2067674080 shutdown_connections 2024-09-21T03:27:04.254 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.251+0000 7f6bf98fc640 1 -- 172.21.15.27:0/2067674080 wait complete. 2024-09-21T03:27:04.256 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.252+0000 7f6bf98fc640 1 Processor -- start 2024-09-21T03:27:04.256 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.252+0000 7f6bf98fc640 1 -- start start 2024-09-21T03:27:04.256 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf98fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bf4070830 0x7f6bf406e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf98fc640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bf4071150 0x7f6bf406edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf98fc640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bf4122310 0x7f6bf41c3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf98fc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf4133730 con 0x7f6bf4070830 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf98fc640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf41335b0 con 0x7f6bf4122310 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf98fc640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f6bf41338b0 con 0x7f6bf4071150 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf27fc640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bf4071150 0x7f6bf406edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf2ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bf4070830 0x7f6bf406e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf27fc640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bf4071150 0x7f6bf406edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:54732/0 (socket says 172.21.15.27:54732) 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf27fc640 1 -- 172.21.15.27:0/217025485 learned_addr learned my addr 172.21.15.27:0/217025485 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf37fe640 1 --2- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bf4122310 0x7f6bf41c3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf27fc640 1 -- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bf4122310 msgr2=0x7f6bf41c3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf27fc640 1 --2- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bf4122310 0x7f6bf41c3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf27fc640 1 -- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bf4070830 msgr2=0x7f6bf406e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:04.257 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf27fc640 1 --2- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bf4070830 0x7f6bf406e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:04.258 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf27fc640 1 -- 172.21.15.27:0/217025485 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf406ff80 con 0x7f6bf4071150 2024-09-21T03:27:04.258 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.253+0000 7f6bf2ffd640 1 --2- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bf4070830 0x7f6bf406e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:04.258 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.254+0000 7f6bf27fc640 1 --2- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bf4071150 0x7f6bf406edd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009780 tx=0x7f6be8033b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:04.258 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.254+0000 7f6bdbfff640 1 -- 172.21.15.27:0/217025485 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be803f020 con 0x7f6bf4071150 2024-09-21T03:27:04.258 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.254+0000 7f6bdbfff640 1 -- 172.21.15.27:0/217025485 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6be80044c0 con 0x7f6bf4071150 2024-09-21T03:27:04.258 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.254+0000 7f6bf98fc640 1 -- 172.21.15.27:0/217025485 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf4130750 con 0x7f6bf4071150 2024-09-21T03:27:04.258 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.254+0000 7f6bdbfff640 1 -- 172.21.15.27:0/217025485 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8037040 con 0x7f6bf4071150 2024-09-21T03:27:04.258 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.254+0000 7f6bf98fc640 1 -- 172.21.15.27:0/217025485 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf406f310 con 0x7f6bf4071150 2024-09-21T03:27:04.259 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.255+0000 7f6bf98fc640 1 -- 172.21.15.27:0/217025485 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf41300c0 con 0x7f6bf4071150 2024-09-21T03:27:04.259 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.255+0000 7f6bdbfff640 1 -- 172.21.15.27:0/217025485 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 4) ==== 55943+0+0 (secure 0 0 0) 0x7f6be8042020 con 0x7f6bf4071150 2024-09-21T03:27:04.259 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.256+0000 7f6bdbfff640 1 -- 172.21.15.27:0/217025485 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(8..8 src has 1..8) ==== 1465+0+0 (secure 0 0 0) 0x7f6be807ceb0 con 0x7f6bf4071150 2024-09-21T03:27:04.262 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.260+0000 7f6bdbfff640 1 -- 172.21.15.27:0/217025485 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f6be8049070 con 0x7f6bf4071150 2024-09-21T03:27:04.430 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.428+0000 7f6bf98fc640 1 -- 172.21.15.27:0/217025485 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6bf4070c10 con 0x7f6bf4071150 2024-09-21T03:27:04.431 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.429+0000 7f6bdbfff640 1 -- 172.21.15.27:0/217025485 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+4153 (secure 0 0 0) 0x7f6be807c130 con 0x7f6bf4071150 2024-09-21T03:27:04.431 INFO:teuthology.misc.health.smithi027.stdout: 2024-09-21T03:27:04.431 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":8,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:27:03.525129+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:27:04.433 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.430+0000 7f6bf98fc640 1 -- 172.21.15.27:0/217025485 >> v1:172.21.15.27:6800/1098903051 conn(0x7f6be0047f40 legacy=0x7f6be004a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:04.433 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.430+0000 7f6bf98fc640 1 -- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bf4071150 msgr2=0x7f6bf406edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:04.433 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.430+0000 7f6bf98fc640 1 --2- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bf4071150 0x7f6bf406edd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6be8009780 tx=0x7f6be8033b70 comp rx=0 tx=0).stop 2024-09-21T03:27:04.433 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.431+0000 7f6bf98fc640 1 -- 172.21.15.27:0/217025485 shutdown_connections 2024-09-21T03:27:04.433 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.431+0000 7f6bf98fc640 1 --2- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bf4122310 0x7f6bf41c3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:04.433 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.431+0000 7f6bf98fc640 1 --2- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bf4071150 0x7f6bf406edd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:04.434 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.431+0000 7f6bf98fc640 1 --2- 172.21.15.27:0/217025485 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bf4070830 0x7f6bf406e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:04.434 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.431+0000 7f6bf98fc640 1 -- 172.21.15.27:0/217025485 >> 172.21.15.27:0/217025485 conn(0x7f6bf409c420 msgr2=0x7f6bf4128340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:04.434 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.431+0000 7f6bf98fc640 1 -- 172.21.15.27:0/217025485 shutdown_connections 2024-09-21T03:27:04.434 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:04.431+0000 7f6bf98fc640 1 -- 172.21.15.27:0/217025485 wait complete. 2024-09-21T03:27:04.442 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-21T03:27:04.442 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-09-21T03:27:04.440+0000 7efd18631640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T03:27:04.445 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-09-21T03:27:04.443+0000 7efd0fc2c640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T03:27:06.228 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-09-21T03:27:06.226+0000 7fb8cde7c640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T03:27:06.236 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-09-21T03:27:06.234+0000 7fb8c64ae640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T03:27:06.262 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-21T03:27:06.260+0000 7f3a0d6c4640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T03:27:06.268 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-21T03:27:06.266+0000 7f3a04cbf640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T03:27:06.286 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-09-21T03:27:06.284+0000 7fbf4c0a2640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T03:27:06.291 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-09-21T03:27:06.289+0000 7fbf4369d640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T03:27:10.845 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T03:27:10.946 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.943+0000 7f4325c53640 1 -- 172.21.15.27:0/1813614592 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f43201337c0 msgr2=0x7f4320135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:10.946 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.943+0000 7f4325c53640 1 --2- 172.21.15.27:0/1813614592 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f43201337c0 0x7f4320135bb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4318009be0 tx=0x7f431802a770 comp rx=0 tx=0).stop 2024-09-21T03:27:10.946 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.943+0000 7f4325c53640 1 -- 172.21.15.27:0/1813614592 shutdown_connections 2024-09-21T03:27:10.947 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.943+0000 7f4325c53640 1 --2- 172.21.15.27:0/1813614592 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f43201337c0 0x7f4320135bb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:10.947 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.943+0000 7f4325c53640 1 --2- 172.21.15.27:0/1813614592 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f4320072ac0 0x7f4320133280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:10.947 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.943+0000 7f4325c53640 1 -- 172.21.15.27:0/1813614592 >> 172.21.15.27:0/1813614592 conn(0x7f43200a0870 msgr2=0x7f43200a0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:10.947 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.944+0000 7f4325c53640 1 -- 172.21.15.27:0/1813614592 shutdown_connections 2024-09-21T03:27:10.947 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.944+0000 7f4325c53640 1 -- 172.21.15.27:0/1813614592 wait complete. 2024-09-21T03:27:10.951 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.944+0000 7f4325c53640 1 Processor -- start 2024-09-21T03:27:10.951 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f4325c53640 1 -- start start 2024-09-21T03:27:10.951 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f4325c53640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4320072ac0 0x7f432012f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:10.951 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f4325c53640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f432012ae80 0x7f432012f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:10.951 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f4325c53640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f43201337c0 0x7f432012fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:10.951 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f4325c53640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f4320136bf0 con 0x7f4320072ac0 2024-09-21T03:27:10.952 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f4325c53640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f4320136a70 con 0x7f432012ae80 2024-09-21T03:27:10.952 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f4325c53640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f4320136d70 con 0x7f43201337c0 2024-09-21T03:27:10.952 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f431f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4320072ac0 0x7f432012f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:10.952 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f431effd640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f432012ae80 0x7f432012f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:10.952 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f431effd640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f432012ae80 0x7f432012f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:43222/0 (socket says 172.21.15.27:43222) 2024-09-21T03:27:10.952 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f431f7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4320072ac0 0x7f432012f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40962/0 (socket says 172.21.15.27:40962) 2024-09-21T03:27:10.953 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f431effd640 1 -- 172.21.15.27:0/1748299056 learned_addr learned my addr 172.21.15.27:0/1748299056 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:10.953 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f431ffff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f43201337c0 0x7f432012fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:10.953 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f431effd640 1 -- 172.21.15.27:0/1748299056 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f43201337c0 msgr2=0x7f432012fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:10.953 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f431effd640 1 --2- 172.21.15.27:0/1748299056 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f43201337c0 0x7f432012fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:10.953 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f431effd640 1 -- 172.21.15.27:0/1748299056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4320072ac0 msgr2=0x7f432012f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:10.953 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f431effd640 1 --2- 172.21.15.27:0/1748299056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4320072ac0 0x7f432012f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:10.953 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.945+0000 7f431effd640 1 -- 172.21.15.27:0/1748299056 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43201329c0 con 0x7f432012ae80 2024-09-21T03:27:10.954 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.946+0000 7f431effd640 1 --2- 172.21.15.27:0/1748299056 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f432012ae80 0x7f432012f730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f430800ea60 tx=0x7f430800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:10.954 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.946+0000 7f4317fff640 1 -- 172.21.15.27:0/1748299056 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430800cde0 con 0x7f432012ae80 2024-09-21T03:27:10.954 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.946+0000 7f4317fff640 1 -- 172.21.15.27:0/1748299056 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4308004020 con 0x7f432012ae80 2024-09-21T03:27:10.954 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.946+0000 7f4325c53640 1 -- 172.21.15.27:0/1748299056 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4320130a70 con 0x7f432012ae80 2024-09-21T03:27:10.954 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.946+0000 7f4317fff640 1 -- 172.21.15.27:0/1748299056 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4308004450 con 0x7f432012ae80 2024-09-21T03:27:10.954 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.946+0000 7f4325c53640 1 -- 172.21.15.27:0/1748299056 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4320131670 con 0x7f432012ae80 2024-09-21T03:27:10.955 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.947+0000 7f4317fff640 1 -- 172.21.15.27:0/1748299056 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f43080035f0 con 0x7f432012ae80 2024-09-21T03:27:10.955 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.947+0000 7f4325c53640 1 -- 172.21.15.27:0/1748299056 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43201344e0 con 0x7f432012ae80 2024-09-21T03:27:10.955 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.947+0000 7f4317fff640 1 -- 172.21.15.27:0/1748299056 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 2459+0+0 (secure 0 0 0) 0x7f430805e160 con 0x7f432012ae80 2024-09-21T03:27:10.955 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:10.951+0000 7f4317fff640 1 -- 172.21.15.27:0/1748299056 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f43201344e0 con 0x7f432012ae80 2024-09-21T03:27:11.104 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.102+0000 7f4325c53640 1 -- 172.21.15.27:0/1748299056 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f432012b260 con 0x7f432012ae80 2024-09-21T03:27:11.105 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.103+0000 7f4317fff640 1 -- 172.21.15.27:0/1748299056 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+7010 (secure 0 0 0) 0x7f430805d450 con 0x7f432012ae80 2024-09-21T03:27:11.105 INFO:teuthology.misc.health.smithi027.stdout: 2024-09-21T03:27:11.105 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":14,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:27:10.245630+0000","last_up_change":"2024-09-21T03:27:07.231513+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6809","nonce":831015510}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6810","nonce":831015510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6812","nonce":831015510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6811","nonce":831015510}]},"public_addr":"172.21.15.27:6809/831015510","cluster_addr":"172.21.15.27:6810/831015510","heartbeat_back_addr":"172.21.15.27:6812/831015510","heartbeat_front_addr":"172.21.15.27:6811/831015510","state":["exists","up"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":1590434734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":1590434734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":1590434734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":1590434734}]},"public_addr":"172.21.15.27:6805/1590434734","cluster_addr":"172.21.15.27:6806/1590434734","heartbeat_back_addr":"172.21.15.27:6808/1590434734","heartbeat_front_addr":"172.21.15.27:6807/1590434734","state":["exists","up"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":3010908863}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":3010908863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":3010908863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":3010908863}]},"public_addr":"172.21.15.27:6801/3010908863","cluster_addr":"172.21.15.27:6802/3010908863","heartbeat_back_addr":"172.21.15.27:6804/3010908863","heartbeat_front_addr":"172.21.15.27:6803/3010908863","state":["exists","up"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1661572621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1661572621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1661572621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1661572621}]},"public_addr":"172.21.15.27:6813/1661572621","cluster_addr":"172.21.15.27:6814/1661572621","heartbeat_back_addr":"172.21.15.27:6816/1661572621","heartbeat_front_addr":"172.21.15.27:6815/1661572621","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:27:11.107 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.105+0000 7f4325c53640 1 -- 172.21.15.27:0/1748299056 >> v1:172.21.15.27:6800/1098903051 conn(0x7f4300047fd0 legacy=0x7f430004a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:11.107 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.105+0000 7f4325c53640 1 -- 172.21.15.27:0/1748299056 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f432012ae80 msgr2=0x7f432012f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:11.108 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.105+0000 7f4325c53640 1 --2- 172.21.15.27:0/1748299056 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f432012ae80 0x7f432012f730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f430800ea60 tx=0x7f430800ef20 comp rx=0 tx=0).stop 2024-09-21T03:27:11.108 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.105+0000 7f4325c53640 1 -- 172.21.15.27:0/1748299056 shutdown_connections 2024-09-21T03:27:11.108 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.105+0000 7f4325c53640 1 --2- 172.21.15.27:0/1748299056 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f43201337c0 0x7f432012fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:11.108 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.105+0000 7f4325c53640 1 --2- 172.21.15.27:0/1748299056 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f432012ae80 0x7f432012f730 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:11.108 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.105+0000 7f4325c53640 1 --2- 172.21.15.27:0/1748299056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4320072ac0 0x7f432012f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:11.108 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.105+0000 7f4325c53640 1 -- 172.21.15.27:0/1748299056 >> 172.21.15.27:0/1748299056 conn(0x7f43200a0870 msgr2=0x7f432012bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:11.108 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.105+0000 7f4325c53640 1 -- 172.21.15.27:0/1748299056 shutdown_connections 2024-09-21T03:27:11.108 INFO:teuthology.misc.health.smithi027.stderr:2024-09-21T03:27:11.105+0000 7f4325c53640 1 -- 172.21.15.27:0/1748299056 wait complete. 2024-09-21T03:27:11.116 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-21T03:27:11.117 INFO:tasks.ceph:Creating RBD pool 2024-09-21T03:27:11.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-21T03:27:11.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.218+0000 7fc614c24640 1 -- 172.21.15.27:0/266749229 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc61012dca0 msgr2=0x7fc61012e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:11.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.218+0000 7fc614c24640 1 --2- 172.21.15.27:0/266749229 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc61012dca0 0x7fc61012e220 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc600009890 tx=0x7fc60002a4a0 comp rx=0 tx=0).stop 2024-09-21T03:27:11.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.218+0000 7fc614c24640 1 -- 172.21.15.27:0/266749229 shutdown_connections 2024-09-21T03:27:11.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.218+0000 7fc614c24640 1 --2- 172.21.15.27:0/266749229 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc61012e760 0x7fc61012a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:11.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.218+0000 7fc614c24640 1 --2- 172.21.15.27:0/266749229 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc61012dca0 0x7fc61012e220 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:11.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.218+0000 7fc614c24640 1 -- 172.21.15.27:0/266749229 >> 172.21.15.27:0/266749229 conn(0x7fc6100a05b0 msgr2=0x7fc6100a09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:11.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.218+0000 7fc614c24640 1 -- 172.21.15.27:0/266749229 shutdown_connections 2024-09-21T03:27:11.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.218+0000 7fc614c24640 1 -- 172.21.15.27:0/266749229 wait complete. 2024-09-21T03:27:11.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc614c24640 1 Processor -- start 2024-09-21T03:27:11.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc614c24640 1 -- start start 2024-09-21T03:27:11.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc614c24640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc61012ace0 0x7fc61012ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:11.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc614c24640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc61012dca0 0x7fc61012f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:11.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc614c24640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61012e760 0x7fc61012f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:11.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc614c24640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc610136aa0 con 0x7fc61012e760 2024-09-21T03:27:11.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc614c24640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fc610136920 con 0x7fc61012ace0 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc614c24640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fc610136c20 con 0x7fc61012dca0 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc60e59c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc61012ace0 0x7fc61012ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc60dd9b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc61012dca0 0x7fc61012f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc60e59c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc61012ace0 0x7fc61012ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:43242/0 (socket says 172.21.15.27:43242) 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc60ed9d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61012e760 0x7fc61012f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.219+0000 7fc60e59c640 1 -- 172.21.15.27:0/2890072959 learned_addr learned my addr 172.21.15.27:0/2890072959 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.220+0000 7fc60dd9b640 1 -- 172.21.15.27:0/2890072959 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc61012ace0 msgr2=0x7fc61012ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.220+0000 7fc60dd9b640 1 --2- 172.21.15.27:0/2890072959 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc61012ace0 0x7fc61012ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.220+0000 7fc60dd9b640 1 -- 172.21.15.27:0/2890072959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61012e760 msgr2=0x7fc61012f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.220+0000 7fc60dd9b640 1 --2- 172.21.15.27:0/2890072959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61012e760 0x7fc61012f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.220+0000 7fc60dd9b640 1 -- 172.21.15.27:0/2890072959 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc610129650 con 0x7fc61012dca0 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.220+0000 7fc60dd9b640 1 --2- 172.21.15.27:0/2890072959 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc61012dca0 0x7fc61012f240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc60400e9c0 tx=0x7fc60400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:11.224 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.220+0000 7fc5feffd640 1 -- 172.21.15.27:0/2890072959 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60400cd20 con 0x7fc61012dca0 2024-09-21T03:27:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.220+0000 7fc5feffd640 1 -- 172.21.15.27:0/2890072959 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc604010430 con 0x7fc61012dca0 2024-09-21T03:27:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.220+0000 7fc614c24640 1 -- 172.21.15.27:0/2890072959 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6101305d0 con 0x7fc61012dca0 2024-09-21T03:27:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.220+0000 7fc5feffd640 1 -- 172.21.15.27:0/2890072959 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc604010860 con 0x7fc61012dca0 2024-09-21T03:27:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.220+0000 7fc614c24640 1 -- 172.21.15.27:0/2890072959 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6101c9e20 con 0x7fc61012dca0 2024-09-21T03:27:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.221+0000 7fc5feffd640 1 -- 172.21.15.27:0/2890072959 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7fc60401f020 con 0x7fc61012dca0 2024-09-21T03:27:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.221+0000 7fc614c24640 1 -- 172.21.15.27:0/2890072959 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc61012e220 con 0x7fc61012dca0 2024-09-21T03:27:11.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.221+0000 7fc5feffd640 1 -- 172.21.15.27:0/2890072959 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(14..14 src has 1..14) ==== 2459+0+0 (secure 0 0 0) 0x7fc60405de30 con 0x7fc61012dca0 2024-09-21T03:27:11.227 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.226+0000 7fc5feffd640 1 -- 172.21.15.27:0/2890072959 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7fc60405d120 con 0x7fc61012dca0 2024-09-21T03:27:11.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:11.377+0000 7fc614c24640 1 -- 172.21.15.27:0/2890072959 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fc610092140 con 0x7fc61012dca0 2024-09-21T03:27:12.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.263+0000 7fc5feffd640 1 -- 172.21.15.27:0/2890072959 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) ==== 109+0+0 (secure 0 0 0) 0x7fc60403d290 con 0x7fc61012dca0 2024-09-21T03:27:12.265 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'rbd' created 2024-09-21T03:27:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.265+0000 7fc614c24640 1 -- 172.21.15.27:0/2890072959 >> v1:172.21.15.27:6800/1098903051 conn(0x7fc5f0047fd0 legacy=0x7fc5f004a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.265+0000 7fc614c24640 1 -- 172.21.15.27:0/2890072959 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc61012dca0 msgr2=0x7fc61012f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.265+0000 7fc614c24640 1 --2- 172.21.15.27:0/2890072959 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc61012dca0 0x7fc61012f240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc60400e9c0 tx=0x7fc60400ee80 comp rx=0 tx=0).stop 2024-09-21T03:27:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.265+0000 7fc614c24640 1 -- 172.21.15.27:0/2890072959 shutdown_connections 2024-09-21T03:27:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.265+0000 7fc614c24640 1 --2- 172.21.15.27:0/2890072959 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc61012e760 0x7fc61012f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.265+0000 7fc614c24640 1 --2- 172.21.15.27:0/2890072959 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc61012dca0 0x7fc61012f240 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.266+0000 7fc614c24640 1 --2- 172.21.15.27:0/2890072959 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc61012ace0 0x7fc61012ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.266+0000 7fc614c24640 1 -- 172.21.15.27:0/2890072959 >> 172.21.15.27:0/2890072959 conn(0x7fc6100a05b0 msgr2=0x7fc61006dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.266+0000 7fc614c24640 1 -- 172.21.15.27:0/2890072959 shutdown_connections 2024-09-21T03:27:12.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.266+0000 7fc614c24640 1 -- 172.21.15.27:0/2890072959 wait complete. 2024-09-21T03:27:12.280 DEBUG:teuthology.orchestra.run.smithi027:> rbd --cluster ceph pool init rbd 2024-09-21T03:27:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.331+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1477561957 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x5635eb635420 msgr2=0x5635eb6c9280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:12.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.331+0000 7f49f1c36d80 1 --2- 172.21.15.27:0/1477561957 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x5635eb635420 0x5635eb6c9280 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f49d800a3d0 tx=0x7f49d80460d0 comp rx=0 tx=0).stop 2024-09-21T03:27:12.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.331+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1477561957 shutdown_connections 2024-09-21T03:27:12.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.331+0000 7f49f1c36d80 1 --2- 172.21.15.27:0/1477561957 >> v2:172.21.15.27:3300/0 conn(0x5635eb6cdce0 0x5635eb6d00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:12.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.331+0000 7f49f1c36d80 1 --2- 172.21.15.27:0/1477561957 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x5635eb635420 0x5635eb6c9280 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:12.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.331+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1477561957 >> 172.21.15.27:0/1477561957 conn(0x5635eb637f40 msgr2=0x5635eb638350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:12.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.331+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1477561957 shutdown_connections 2024-09-21T03:27:12.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.331+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1477561957 wait complete. 2024-09-21T03:27:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f1c36d80 1 Processor -- start 2024-09-21T03:27:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f1c36d80 1 -- start start 2024-09-21T03:27:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f1c36d80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635eb635420 0x5635eb6c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f1c36d80 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x5635eb6c8790 0x5635eb6ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f1c36d80 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x5635eb6cdce0 0x5635eb6ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f1c36d80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x5635eb6d0ca0 con 0x5635eb635420 2024-09-21T03:27:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f1c36d80 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x5635eb4b5540 con 0x5635eb6c8790 2024-09-21T03:27:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f1c36d80 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x5635eb6d0e20 con 0x5635eb6cdce0 2024-09-21T03:27:12.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f072e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635eb635420 0x5635eb6c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49e7f2d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x5635eb6c8790 0x5635eb6ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f1990640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x5635eb6cdce0 0x5635eb6ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f072e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635eb635420 0x5635eb6c9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40578/0 (socket says 172.21.15.27:40578) 2024-09-21T03:27:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.332+0000 7f49f072e640 1 -- 172.21.15.27:0/1726753432 learned_addr learned my addr 172.21.15.27:0/1726753432 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49f1990640 1 -- 172.21.15.27:0/1726753432 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x5635eb6c8790 msgr2=0x5635eb6ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49f1990640 1 --2- 172.21.15.27:0/1726753432 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x5635eb6c8790 0x5635eb6ca470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49f1990640 1 -- 172.21.15.27:0/1726753432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635eb635420 msgr2=0x5635eb6c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49f1990640 1 --2- 172.21.15.27:0/1726753432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635eb635420 0x5635eb6c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49f1990640 1 -- 172.21.15.27:0/1726753432 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635eb6cc840 con 0x5635eb6cdce0 2024-09-21T03:27:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49e7f2d640 1 --2- 172.21.15.27:0/1726753432 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x5635eb6c8790 0x5635eb6ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:12.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49f072e640 1 --2- 172.21.15.27:0/1726753432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5635eb635420 0x5635eb6c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:12.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49f1990640 1 --2- 172.21.15.27:0/1726753432 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x5635eb6cdce0 0x5635eb6ca9b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f49e800d630 tx=0x7f49e800daf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:12.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e800f900 con 0x5635eb6cdce0 2024-09-21T03:27:12.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49e800faa0 con 0x5635eb6cdce0 2024-09-21T03:27:12.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x5635eb6d95b0 con 0x5635eb6cdce0 2024-09-21T03:27:12.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e800e420 con 0x5635eb6cdce0 2024-09-21T03:27:12.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.333+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x5635eb6d9b00 con 0x5635eb6cdce0 2024-09-21T03:27:12.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.334+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f49e800e5c0 con 0x5635eb6cdce0 2024-09-21T03:27:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.335+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(15..15 src has 1..15) ==== 2801+0+0 (secure 0 0 0) 0x7f49e805d230 con 0x5635eb6cdce0 2024-09-21T03:27:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:12.335+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x5635eb6d9fa0 con 0x5635eb6cdce0 2024-09-21T03:27:13.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.265+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v16) ==== 141+0+0 (secure 0 0 0) 0x5635eb6cc840 con 0x5635eb6cdce0 2024-09-21T03:27:13.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.266+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f49d8051070 con 0x5635eb6cdce0 2024-09-21T03:27:13.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.266+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f49e803a530 con 0x5635eb6cdce0 2024-09-21T03:27:13.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.266+0000 7f49ef72c640 1 -- 172.21.15.27:0/1726753432 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=16}) -- 0x7f49d4002ae0 con 0x5635eb6cdce0 2024-09-21T03:27:13.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.266+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 8 ==== osd_map(16..16 src has 1..16) ==== 650+0+0 (secure 0 0 0) 0x7f49e805c3c0 con 0x5635eb6cdce0 2024-09-21T03:27:13.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.266+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 --> v1:172.21.15.27:6813/1661572621 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) -- 0x5635eb6e4db0 con 0x5635eb6e12a0 2024-09-21T03:27:13.285 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.283+0000 7f49e7f2d640 1 -- 172.21.15.27:0/1726753432 <== osd.3 v1:172.21.15.27:6813/1661572621 1 ==== osd_op_reply(1 rbd_trash [create] v16'1 uv1 ondisk = 0) ==== 153+0+0 (unknown 2685446591 0 0) 0x7f49d804d760 con 0x5635eb6e12a0 2024-09-21T03:27:13.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.285+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 --> v1:172.21.15.27:6813/1661572621 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) -- 0x5635eb736e90 con 0x5635eb6e12a0 2024-09-21T03:27:13.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.286+0000 7f49e7f2d640 1 -- 172.21.15.27:0/1726753432 <== osd.3 v1:172.21.15.27:6813/1661572621 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 2026326479 0 0) 0x7f49d804d760 con 0x5635eb6e12a0 2024-09-21T03:27:13.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.286+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 --> v1:172.21.15.27:6813/1661572621 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) -- 0x5635eb735f80 con 0x5635eb6e12a0 2024-09-21T03:27:13.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.286+0000 7f49e7f2d640 1 -- 172.21.15.27:0/1726753432 <== osd.3 v1:172.21.15.27:6813/1661572621 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 2039985091 0 0) 0x7f49d804d760 con 0x5635eb6e12a0 2024-09-21T03:27:13.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:13.286+0000 7f49ef72c640 1 -- 172.21.15.27:0/1726753432 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f49d4003ef0 con 0x5635eb6cdce0 2024-09-21T03:27:14.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:14.270+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 9 ==== pool_op_reply(tid 4 (0) Success v17) ==== 55+0+0 (secure 0 0 0) 0x7f49e8013070 con 0x5635eb6cdce0 2024-09-21T03:27:14.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:14.270+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=17}) -- 0x7f49d004d450 con 0x5635eb6cdce0 2024-09-21T03:27:14.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:14.271+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 10 ==== osd_map(17..17 src has 1..17) ==== 626+0+0 (secure 0 0 0) 0x7f49e803c500 con 0x5635eb6cdce0 2024-09-21T03:27:14.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:14.271+0000 7f49ef72c640 1 -- 172.21.15.27:0/1726753432 --> v1:172.21.15.27:6813/1661572621 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x7f49d40058f0 con 0x5635eb6e12a0 2024-09-21T03:27:14.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:14.274+0000 7f49e7f2d640 1 -- 172.21.15.27:0/1726753432 <== osd.3 v1:172.21.15.27:6813/1661572621 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v17'1 uv1 ondisk = 0) ==== 194+0+0 (unknown 3115756469 0 0) 0x7f49d804d760 con 0x5635eb6e12a0 2024-09-21T03:27:14.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:14.274+0000 7f49ef72c640 1 -- 172.21.15.27:0/1726753432 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v17) -- 0x7f49d40053c0 con 0x5635eb6cdce0 2024-09-21T03:27:15.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.272+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 11 ==== pool_op_reply(tid 6 (0) Success v18) ==== 43+0+0 (secure 0 0 0) 0x7f49e803c760 con 0x5635eb6cdce0 2024-09-21T03:27:15.274 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.272+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=18}) -- 0x7f49d004d6d0 con 0x5635eb6cdce0 2024-09-21T03:27:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.275+0000 7f49ed728640 1 -- 172.21.15.27:0/1726753432 <== mon.2 v2:172.21.15.27:3302/0 12 ==== osd_map(18..18 src has 1..18) ==== 654+0+0 (secure 0 0 0) 0x7f49e803aad0 con 0x5635eb6cdce0 2024-09-21T03:27:15.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.275+0000 7f49ef72c640 1 -- 172.21.15.27:0/1726753432 --> v1:172.21.15.27:6813/1661572621 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f49d4006140 con 0x5635eb6e12a0 2024-09-21T03:27:15.287 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.285+0000 7f49e7f2d640 1 -- 172.21.15.27:0/1726753432 <== osd.3 v1:172.21.15.27:6813/1661572621 5 ==== osd_op_reply(7 rbd_info [write 0~19] v18'2 uv2 ondisk = 0) ==== 152+0+0 (unknown 4030416863 0 0) 0x7f49d804d760 con 0x5635eb6e12a0 2024-09-21T03:27:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.285+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 >> v1:172.21.15.27:6813/1661572621 conn(0x5635eb6e12a0 legacy=0x5635eb6e3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.286+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 >> v1:172.21.15.27:6800/1098903051 conn(0x7f49d00468c0 legacy=0x7f49d0048d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.286+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x5635eb6cdce0 msgr2=0x5635eb6ca9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.286+0000 7f49f1c36d80 1 --2- 172.21.15.27:0/1726753432 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x5635eb6cdce0 0x5635eb6ca9b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f49e800d630 tx=0x7f49e800daf0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.286+0000 7f49f1990640 1 -- 172.21.15.27:0/1726753432 reap_dead start 2024-09-21T03:27:15.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.286+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 shutdown_connections 2024-09-21T03:27:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.286+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 >> 172.21.15.27:0/1726753432 conn(0x5635eb637f40 msgr2=0x5635eb6361e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.286+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 shutdown_connections 2024-09-21T03:27:15.289 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.286+0000 7f49f1c36d80 1 -- 172.21.15.27:0/1726753432 wait complete. 2024-09-21T03:27:15.292 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-21T03:27:15.293 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-21T03:27:15.293 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T03:27:15.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.401+0000 7fbed7044640 1 -- 172.21.15.27:0/2180459859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed012dd40 msgr2=0x7fbed012e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.401+0000 7fbed7044640 1 --2- 172.21.15.27:0/2180459859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed012dd40 0x7fbed012e3e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbec0009830 tx=0x7fbec002a3f0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.401+0000 7fbed7044640 1 -- 172.21.15.27:0/2180459859 shutdown_connections 2024-09-21T03:27:15.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.401+0000 7fbed7044640 1 --2- 172.21.15.27:0/2180459859 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbed012e920 0x7fbed012aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.401+0000 7fbed7044640 1 --2- 172.21.15.27:0/2180459859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed012dd40 0x7fbed012e3e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.401+0000 7fbed7044640 1 -- 172.21.15.27:0/2180459859 >> 172.21.15.27:0/2180459859 conn(0x7fbed00a04c0 msgr2=0x7fbed00a08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:15.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.401+0000 7fbed7044640 1 -- 172.21.15.27:0/2180459859 shutdown_connections 2024-09-21T03:27:15.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.401+0000 7fbed7044640 1 -- 172.21.15.27:0/2180459859 wait complete. 2024-09-21T03:27:15.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbed7044640 1 Processor -- start 2024-09-21T03:27:15.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbed7044640 1 -- start start 2024-09-21T03:27:15.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbed7044640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbed0099be0 0x7fbed00708e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbed7044640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbed012dd40 0x7fbed0070e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbed7044640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed012e920 0x7fbed0071360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbed7044640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbed0136c40 con 0x7fbed012e920 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbed7044640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fbed0136ac0 con 0x7fbed012dd40 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbed7044640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fbed0136dc0 con 0x7fbed0099be0 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbed4de0640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbed0099be0 0x7fbed00708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbecffff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbed012dd40 0x7fbed0070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbed55e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed012e920 0x7fbed0071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.402+0000 7fbed4de0640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbed0099be0 0x7fbed00708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:46978/0 (socket says 172.21.15.27:46978) 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbed4de0640 1 -- 172.21.15.27:0/1630798640 learned_addr learned my addr 172.21.15.27:0/1630798640 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbed4de0640 1 -- 172.21.15.27:0/1630798640 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbed012dd40 msgr2=0x7fbed0070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbed4de0640 1 --2- 172.21.15.27:0/1630798640 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbed012dd40 0x7fbed0070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbed4de0640 1 -- 172.21.15.27:0/1630798640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed012e920 msgr2=0x7fbed0071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbed4de0640 1 --2- 172.21.15.27:0/1630798640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed012e920 0x7fbed0071360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbed4de0640 1 -- 172.21.15.27:0/1630798640 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed0072760 con 0x7fbed0099be0 2024-09-21T03:27:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbed4de0640 1 --2- 172.21.15.27:0/1630798640 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbed0099be0 0x7fbed00708e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbec002a9e0 tx=0x7fbec0032c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbecd7fa640 1 -- 172.21.15.27:0/1630798640 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec0040070 con 0x7fbed0099be0 2024-09-21T03:27:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbecd7fa640 1 -- 172.21.15.27:0/1630798640 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbec0005310 con 0x7fbed0099be0 2024-09-21T03:27:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbed7044640 1 -- 172.21.15.27:0/1630798640 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed00730f0 con 0x7fbed0099be0 2024-09-21T03:27:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbecd7fa640 1 -- 172.21.15.27:0/1630798640 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec00047a0 con 0x7fbed0099be0 2024-09-21T03:27:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.403+0000 7fbed7044640 1 -- 172.21.15.27:0/1630798640 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed0072090 con 0x7fbed0099be0 2024-09-21T03:27:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.404+0000 7fbed7044640 1 -- 172.21.15.27:0/1630798640 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed012e3e0 con 0x7fbed0099be0 2024-09-21T03:27:15.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.404+0000 7fbecd7fa640 1 -- 172.21.15.27:0/1630798640 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7fbec0039040 con 0x7fbed0099be0 2024-09-21T03:27:15.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.405+0000 7fbecd7fa640 1 -- 172.21.15.27:0/1630798640 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7fbec007ec40 con 0x7fbed0099be0 2024-09-21T03:27:15.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.409+0000 7fbecd7fa640 1 -- 172.21.15.27:0/1630798640 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7fbed00730f0 con 0x7fbed0099be0 2024-09-21T03:27:15.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.560+0000 7fbed7044640 1 -- 172.21.15.27:0/1630798640 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fbed009c3f0 con 0x7fbed0099be0 2024-09-21T03:27:15.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.561+0000 7fbecd7fa640 1 -- 172.21.15.27:0/1630798640 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v27) ==== 86+0+160 (secure 0 0 0) 0x7fbec007dd90 con 0x7fbed0099be0 2024-09-21T03:27:15.563 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:27:15.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.563+0000 7fbed7044640 1 -- 172.21.15.27:0/1630798640 >> v1:172.21.15.27:6800/1098903051 conn(0x7fbeb0047fd0 legacy=0x7fbeb004a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.563+0000 7fbed7044640 1 -- 172.21.15.27:0/1630798640 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbed0099be0 msgr2=0x7fbed00708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.563+0000 7fbed7044640 1 --2- 172.21.15.27:0/1630798640 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbed0099be0 0x7fbed00708e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbec002a9e0 tx=0x7fbec0032c10 comp rx=0 tx=0).stop 2024-09-21T03:27:15.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.563+0000 7fbed7044640 1 -- 172.21.15.27:0/1630798640 shutdown_connections 2024-09-21T03:27:15.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.563+0000 7fbed7044640 1 --2- 172.21.15.27:0/1630798640 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbed012e920 0x7fbed0071360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.563+0000 7fbed7044640 1 --2- 172.21.15.27:0/1630798640 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbed012dd40 0x7fbed0070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.563+0000 7fbed7044640 1 --2- 172.21.15.27:0/1630798640 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbed0099be0 0x7fbed00708e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.563+0000 7fbed7044640 1 -- 172.21.15.27:0/1630798640 >> 172.21.15.27:0/1630798640 conn(0x7fbed00a04c0 msgr2=0x7fbed0127d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:15.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.563+0000 7fbed7044640 1 -- 172.21.15.27:0/1630798640 shutdown_connections 2024-09-21T03:27:15.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.563+0000 7fbed7044640 1 -- 172.21.15.27:0/1630798640 wait complete. 2024-09-21T03:27:15.576 INFO:teuthology.orchestra.run.smithi027.stdout:[{"version":27,"timestamp":"2024-09-21T03:27:06.286420+0000","name":"","changes":[{"name":"osd.3/osd_mclock_max_capacity_iops_ssd","new_value":"5306.685639"}]}] 2024-09-21T03:27:15.576 INFO:tasks.ceph_manager:config epoch is 27 2024-09-21T03:27:15.576 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T03:27:15.576 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T03:27:15.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T03:27:15.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.677+0000 7f6bbab2c640 1 -- 172.21.15.27:0/3080884936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb412e280 msgr2=0x7f6bb412e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.677+0000 7f6bbab2c640 1 --2- 172.21.15.27:0/3080884936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb412e280 0x7f6bb412e6e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6bac005fd0 tx=0x7f6bac02a780 comp rx=0 tx=0).stop 2024-09-21T03:27:15.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.677+0000 7f6bbab2c640 1 -- 172.21.15.27:0/3080884936 shutdown_connections 2024-09-21T03:27:15.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.677+0000 7f6bbab2c640 1 --2- 172.21.15.27:0/3080884936 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb412e280 0x7f6bb412e6e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.677+0000 7f6bbab2c640 1 --2- 172.21.15.27:0/3080884936 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bb4097eb0 0x7f6bb4098290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.677+0000 7f6bbab2c640 1 -- 172.21.15.27:0/3080884936 >> 172.21.15.27:0/3080884936 conn(0x7f6bb40a0850 msgr2=0x7f6bb40a0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:15.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.677+0000 7f6bbab2c640 1 -- 172.21.15.27:0/3080884936 shutdown_connections 2024-09-21T03:27:15.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.677+0000 7f6bbab2c640 1 -- 172.21.15.27:0/3080884936 wait complete. 2024-09-21T03:27:15.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bbab2c640 1 Processor -- start 2024-09-21T03:27:15.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bbab2c640 1 -- start start 2024-09-21T03:27:15.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bbab2c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bb4097eb0 0x7f6bb41c3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:15.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bbab2c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bb40987d0 0x7f6bb41c4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:15.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bbab2c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb412e280 0x7f6bb4070740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:15.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bbab2c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6bb41368d0 con 0x7f6bb412e280 2024-09-21T03:27:15.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bbab2c640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f6bb4136750 con 0x7f6bb40987d0 2024-09-21T03:27:15.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bbab2c640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f6bb4136a50 con 0x7f6bb4097eb0 2024-09-21T03:27:15.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bb3fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bb40987d0 0x7f6bb41c4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bb88c8640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bb4097eb0 0x7f6bb41c3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bb90c9640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb412e280 0x7f6bb4070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bb88c8640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bb4097eb0 0x7f6bb41c3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:46990/0 (socket says 172.21.15.27:46990) 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.678+0000 7f6bb3fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bb40987d0 0x7f6bb41c4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:56502/0 (socket says 172.21.15.27:56502) 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bb88c8640 1 -- 172.21.15.27:0/908100123 learned_addr learned my addr 172.21.15.27:0/908100123 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bb90c9640 1 -- 172.21.15.27:0/908100123 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bb4097eb0 msgr2=0x7f6bb41c3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bb90c9640 1 --2- 172.21.15.27:0/908100123 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bb4097eb0 0x7f6bb41c3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bb90c9640 1 -- 172.21.15.27:0/908100123 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bb40987d0 msgr2=0x7f6bb41c4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bb90c9640 1 --2- 172.21.15.27:0/908100123 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bb40987d0 0x7f6bb41c4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bb90c9640 1 -- 172.21.15.27:0/908100123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bb40721f0 con 0x7f6bb412e280 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bb3fff640 1 --2- 172.21.15.27:0/908100123 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bb40987d0 0x7f6bb41c4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bb90c9640 1 --2- 172.21.15.27:0/908100123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb412e280 0x7f6bb4070740 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4050240 tx=0x7f6bac003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bb17fa640 1 -- 172.21.15.27:0/908100123 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bac040070 con 0x7f6bb412e280 2024-09-21T03:27:15.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bb17fa640 1 -- 172.21.15.27:0/908100123 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6bac003e50 con 0x7f6bb412e280 2024-09-21T03:27:15.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bbab2c640 1 -- 172.21.15.27:0/908100123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bb41c4a40 con 0x7f6bb412e280 2024-09-21T03:27:15.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bb17fa640 1 -- 172.21.15.27:0/908100123 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bac004800 con 0x7f6bb412e280 2024-09-21T03:27:15.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.679+0000 7f6bbab2c640 1 -- 172.21.15.27:0/908100123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bb4072850 con 0x7f6bb412e280 2024-09-21T03:27:15.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.680+0000 7f6bbab2c640 1 -- 172.21.15.27:0/908100123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bb4098290 con 0x7f6bb412e280 2024-09-21T03:27:15.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.680+0000 7f6bb17fa640 1 -- 172.21.15.27:0/908100123 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f6bac043070 con 0x7f6bb412e280 2024-09-21T03:27:15.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.681+0000 7f6bb17fa640 1 -- 172.21.15.27:0/908100123 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f6bac07eea0 con 0x7f6bb412e280 2024-09-21T03:27:15.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.685+0000 7f6bb17fa640 1 -- 172.21.15.27:0/908100123 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f6bac07dff0 con 0x7f6bb412e280 2024-09-21T03:27:15.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.863+0000 7f6bbab2c640 1 -- 172.21.15.27:0/908100123 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f6bb4067c80 con 0x7f6bb412e280 2024-09-21T03:27:15.867 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.865+0000 7f6bb17fa640 1 -- 172.21.15.27:0/908100123 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+108901 (secure 0 0 0) 0x7f6bac0327a0 con 0x7f6bb412e280 2024-09-21T03:27:15.869 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:27:15.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.868+0000 7f6bbab2c640 1 -- 172.21.15.27:0/908100123 >> v1:172.21.15.27:6800/1098903051 conn(0x7f6b98047fd0 legacy=0x7f6b9804a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.868+0000 7f6bbab2c640 1 -- 172.21.15.27:0/908100123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb412e280 msgr2=0x7f6bb4070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.868+0000 7f6bbab2c640 1 --2- 172.21.15.27:0/908100123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb412e280 0x7f6bb4070740 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4050240 tx=0x7f6bac003ba0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.868+0000 7f6bbab2c640 1 -- 172.21.15.27:0/908100123 shutdown_connections 2024-09-21T03:27:15.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.868+0000 7f6bbab2c640 1 --2- 172.21.15.27:0/908100123 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb412e280 0x7f6bb4070740 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.868+0000 7f6bbab2c640 1 --2- 172.21.15.27:0/908100123 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6bb40987d0 0x7f6bb41c4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.870 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.868+0000 7f6bbab2c640 1 --2- 172.21.15.27:0/908100123 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6bb4097eb0 0x7f6bb41c3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.868+0000 7f6bbab2c640 1 -- 172.21.15.27:0/908100123 >> 172.21.15.27:0/908100123 conn(0x7f6bb40a0850 msgr2=0x7f6bb412af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:15.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.868+0000 7f6bbab2c640 1 -- 172.21.15.27:0/908100123 shutdown_connections 2024-09-21T03:27:15.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.868+0000 7f6bbab2c640 1 -- 172.21.15.27:0/908100123 wait complete. 2024-09-21T03:27:15.882 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6800","nonce":1098903051}]},"active_addr":"172.21.15.27:6800/1098903051","active_change":"2024-09-21T03:26:57.495177+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4099,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1012963840}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3140689679}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1436650201}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3459873364}]}]} 2024-09-21T03:27:15.884 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T03:27:15.884 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T03:27:15.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:27:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.985+0000 7f49be4e5640 1 -- 172.21.15.27:0/1003436216 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f49b80906c0 msgr2=0x7f49b8132e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.985+0000 7f49be4e5640 1 --2- 172.21.15.27:0/1003436216 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f49b80906c0 0x7f49b8132e30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f49b0005fd0 tx=0x7f49b002aa10 comp rx=0 tx=0).stop 2024-09-21T03:27:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.985+0000 7f49be4e5640 1 -- 172.21.15.27:0/1003436216 shutdown_connections 2024-09-21T03:27:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.985+0000 7f49be4e5640 1 --2- 172.21.15.27:0/1003436216 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f49b80906c0 0x7f49b8132e30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.985+0000 7f49be4e5640 1 --2- 172.21.15.27:0/1003436216 >> v2:172.21.15.27:3300/0 conn(0x7f49b80920d0 0x7f49b80924b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.985+0000 7f49be4e5640 1 -- 172.21.15.27:0/1003436216 >> 172.21.15.27:0/1003436216 conn(0x7f49b809d760 msgr2=0x7f49b8099c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.985+0000 7f49be4e5640 1 -- 172.21.15.27:0/1003436216 shutdown_connections 2024-09-21T03:27:15.988 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.985+0000 7f49be4e5640 1 -- 172.21.15.27:0/1003436216 wait complete. 2024-09-21T03:27:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.986+0000 7f49be4e5640 1 Processor -- start 2024-09-21T03:27:15.990 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.986+0000 7f49be4e5640 1 -- start start 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49be4e5640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f49b80906c0 0x7f49b8073210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49be4e5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b80920d0 0x7f49b8073750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49be4e5640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f49b80929f0 0x7f49b806f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49be4e5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f49b8133700 con 0x7f49b80920d0 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49be4e5640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f49b8133580 con 0x7f49b80929f0 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49be4e5640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f49b8133880 con 0x7f49b80906c0 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49b77fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b80920d0 0x7f49b8073750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49b7fff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f49b80906c0 0x7f49b8073210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49b7fff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f49b80906c0 0x7f49b8073210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:47010/0 (socket says 172.21.15.27:47010) 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49b7fff640 1 -- 172.21.15.27:0/4206416085 learned_addr learned my addr 172.21.15.27:0/4206416085 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49bca82640 1 --2- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f49b80929f0 0x7f49b806f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49b7fff640 1 -- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f49b80929f0 msgr2=0x7f49b806f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49b7fff640 1 --2- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f49b80929f0 0x7f49b806f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49b7fff640 1 -- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b80920d0 msgr2=0x7f49b8073750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:15.991 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49b7fff640 1 --2- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b80920d0 0x7f49b8073750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49b7fff640 1 -- 172.21.15.27:0/4206416085 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b8072a00 con 0x7f49b80906c0 2024-09-21T03:27:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49b77fe640 1 --2- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b80920d0 0x7f49b8073750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.987+0000 7f49b7fff640 1 --2- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f49b80906c0 0x7f49b8073210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f49a800ea20 tx=0x7f49a800eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.988+0000 7f49b4ff9640 1 -- 172.21.15.27:0/4206416085 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a800cd80 con 0x7f49b80906c0 2024-09-21T03:27:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.988+0000 7f49b4ff9640 1 -- 172.21.15.27:0/4206416085 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f49a8010430 con 0x7f49b80906c0 2024-09-21T03:27:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.988+0000 7f49be4e5640 1 -- 172.21.15.27:0/4206416085 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b8070d70 con 0x7f49b80906c0 2024-09-21T03:27:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.988+0000 7f49b4ff9640 1 -- 172.21.15.27:0/4206416085 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a8010840 con 0x7f49b80906c0 2024-09-21T03:27:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.988+0000 7f49be4e5640 1 -- 172.21.15.27:0/4206416085 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b8072030 con 0x7f49b80906c0 2024-09-21T03:27:15.992 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.989+0000 7f49b4ff9640 1 -- 172.21.15.27:0/4206416085 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f49a80109e0 con 0x7f49b80906c0 2024-09-21T03:27:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.989+0000 7f49be4e5640 1 -- 172.21.15.27:0/4206416085 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b8099640 con 0x7f49b80906c0 2024-09-21T03:27:15.993 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.989+0000 7f49b4ff9640 1 -- 172.21.15.27:0/4206416085 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f49a805e480 con 0x7f49b80906c0 2024-09-21T03:27:15.995 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:15.993+0000 7f49b4ff9640 1 -- 172.21.15.27:0/4206416085 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f49b8099640 con 0x7f49b80906c0 2024-09-21T03:27:16.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.141+0000 7f49be4e5640 1 -- 172.21.15.27:0/4206416085 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49b806dcc0 con 0x7f49b80906c0 2024-09-21T03:27:16.144 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.142+0000 7f49b4ff9640 1 -- 172.21.15.27:0/4206416085 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+8909 (secure 0 0 0) 0x7f49b806dcc0 con 0x7f49b80906c0 2024-09-21T03:27:16.144 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:27:16.144 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":18,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:27:15.268896+0000","last_up_change":"2024-09-21T03:27:07.231513+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T03:27:11.408521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6809","nonce":831015510}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6810","nonce":831015510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6812","nonce":831015510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6811","nonce":831015510}]},"public_addr":"172.21.15.27:6809/831015510","cluster_addr":"172.21.15.27:6810/831015510","heartbeat_back_addr":"172.21.15.27:6812/831015510","heartbeat_front_addr":"172.21.15.27:6811/831015510","state":["exists","up"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":1590434734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":1590434734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":1590434734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":1590434734}]},"public_addr":"172.21.15.27:6805/1590434734","cluster_addr":"172.21.15.27:6806/1590434734","heartbeat_back_addr":"172.21.15.27:6808/1590434734","heartbeat_front_addr":"172.21.15.27:6807/1590434734","state":["exists","up"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":3010908863}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":3010908863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":3010908863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":3010908863}]},"public_addr":"172.21.15.27:6801/3010908863","cluster_addr":"172.21.15.27:6802/3010908863","heartbeat_back_addr":"172.21.15.27:6804/3010908863","heartbeat_front_addr":"172.21.15.27:6803/3010908863","state":["exists","up"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1661572621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1661572621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1661572621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1661572621}]},"public_addr":"172.21.15.27:6813/1661572621","cluster_addr":"172.21.15.27:6814/1661572621","heartbeat_back_addr":"172.21.15.27:6816/1661572621","heartbeat_front_addr":"172.21.15.27:6815/1661572621","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:27:16.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.143+0000 7f49be4e5640 1 -- 172.21.15.27:0/4206416085 >> v1:172.21.15.27:6800/1098903051 conn(0x7f49a4047fd0 legacy=0x7f49a404a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.143+0000 7f49be4e5640 1 -- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f49b80906c0 msgr2=0x7f49b8073210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.143+0000 7f49be4e5640 1 --2- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f49b80906c0 0x7f49b8073210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f49a800ea20 tx=0x7f49a800eee0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.144+0000 7f49be4e5640 1 -- 172.21.15.27:0/4206416085 shutdown_connections 2024-09-21T03:27:16.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.144+0000 7f49be4e5640 1 --2- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f49b80929f0 0x7f49b806f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.144+0000 7f49be4e5640 1 --2- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b80920d0 0x7f49b8073750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.144+0000 7f49be4e5640 1 --2- 172.21.15.27:0/4206416085 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f49b80906c0 0x7f49b8073210 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.144+0000 7f49be4e5640 1 -- 172.21.15.27:0/4206416085 >> 172.21.15.27:0/4206416085 conn(0x7f49b809d760 msgr2=0x7f49b80965d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.144+0000 7f49be4e5640 1 -- 172.21.15.27:0/4206416085 shutdown_connections 2024-09-21T03:27:16.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.144+0000 7f49be4e5640 1 -- 172.21.15.27:0/4206416085 wait complete. 2024-09-21T03:27:16.157 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T03:27:16.157 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:27:16.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.257+0000 7f22257c2640 1 -- 172.21.15.27:0/819235387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22200929f0 msgr2=0x7f222008ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.257+0000 7f22257c2640 1 --2- 172.21.15.27:0/819235387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22200929f0 0x7f222008ffb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2214009860 tx=0x7f221402a4b0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.257+0000 7f22257c2640 1 -- 172.21.15.27:0/819235387 shutdown_connections 2024-09-21T03:27:16.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.257+0000 7f22257c2640 1 --2- 172.21.15.27:0/819235387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22200929f0 0x7f222008ffb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.257+0000 7f22257c2640 1 --2- 172.21.15.27:0/819235387 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f22200920d0 0x7f22200924b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.257+0000 7f22257c2640 1 -- 172.21.15.27:0/819235387 >> 172.21.15.27:0/819235387 conn(0x7f222009d760 msgr2=0x7f2220099c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.257+0000 7f22257c2640 1 -- 172.21.15.27:0/819235387 shutdown_connections 2024-09-21T03:27:16.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.257+0000 7f22257c2640 1 -- 172.21.15.27:0/819235387 wait complete. 2024-09-21T03:27:16.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.258+0000 7f22257c2640 1 Processor -- start 2024-09-21T03:27:16.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.258+0000 7f22257c2640 1 -- start start 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f22257c2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22200904f0 0x7f2220073100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f22257c2640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f22200920d0 0x7f2220073640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f22257c2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f22200929f0 0x7f222006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f22257c2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2220133700 con 0x7f22200904f0 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f22257c2640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f2220133580 con 0x7f22200929f0 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f22257c2640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f2220133880 con 0x7f22200920d0 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22200904f0 0x7f2220073100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221effd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22200904f0 0x7f2220073100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40654/0 (socket says 172.21.15.27:40654) 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221effd640 1 -- 172.21.15.27:0/4228896162 learned_addr learned my addr 172.21.15.27:0/4228896162 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221e7fc640 1 --2- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f22200920d0 0x7f2220073640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221f7fe640 1 --2- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f22200929f0 0x7f222006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221effd640 1 -- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f22200920d0 msgr2=0x7f2220073640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.263 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221effd640 1 --2- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f22200920d0 0x7f2220073640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221effd640 1 -- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f22200929f0 msgr2=0x7f222006f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221effd640 1 --2- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f22200929f0 0x7f222006f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221effd640 1 -- 172.21.15.27:0/4228896162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2220073b80 con 0x7f22200904f0 2024-09-21T03:27:16.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221f7fe640 1 --2- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f22200929f0 0x7f222006f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:16.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.259+0000 7f221effd640 1 --2- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22200904f0 0x7f2220073100 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f221000e9f0 tx=0x7f221000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:16.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.260+0000 7f22077fe640 1 -- 172.21.15.27:0/4228896162 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221000cd80 con 0x7f22200904f0 2024-09-21T03:27:16.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.260+0000 7f22077fe640 1 -- 172.21.15.27:0/4228896162 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2210010430 con 0x7f22200904f0 2024-09-21T03:27:16.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.260+0000 7f22257c2640 1 -- 172.21.15.27:0/4228896162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2220072490 con 0x7f22200904f0 2024-09-21T03:27:16.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.260+0000 7f22077fe640 1 -- 172.21.15.27:0/4228896162 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2210010840 con 0x7f22200904f0 2024-09-21T03:27:16.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.260+0000 7f22257c2640 1 -- 172.21.15.27:0/4228896162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2220071bf0 con 0x7f22200904f0 2024-09-21T03:27:16.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.261+0000 7f22257c2640 1 -- 172.21.15.27:0/4228896162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22200924b0 con 0x7f22200904f0 2024-09-21T03:27:16.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.261+0000 7f22077fe640 1 -- 172.21.15.27:0/4228896162 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f22100109e0 con 0x7f22200904f0 2024-09-21T03:27:16.265 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.261+0000 7f22077fe640 1 -- 172.21.15.27:0/4228896162 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f221005e450 con 0x7f22200904f0 2024-09-21T03:27:16.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.265+0000 7f22077fe640 1 -- 172.21.15.27:0/4228896162 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f221005d5a0 con 0x7f22200904f0 2024-09-21T03:27:16.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.416+0000 7f22257c2640 1 -- 172.21.15.27:0/4228896162 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2220067c80 con 0x7f22200904f0 2024-09-21T03:27:16.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.417+0000 7f22077fe640 1 -- 172.21.15.27:0/4228896162 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+8909 (secure 0 0 0) 0x7f221003d7a0 con 0x7f22200904f0 2024-09-21T03:27:16.419 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:27:16.419 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":18,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:27:15.268896+0000","last_up_change":"2024-09-21T03:27:07.231513+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T03:27:11.408521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6809","nonce":831015510}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6810","nonce":831015510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6812","nonce":831015510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6811","nonce":831015510}]},"public_addr":"172.21.15.27:6809/831015510","cluster_addr":"172.21.15.27:6810/831015510","heartbeat_back_addr":"172.21.15.27:6812/831015510","heartbeat_front_addr":"172.21.15.27:6811/831015510","state":["exists","up"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":1590434734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":1590434734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":1590434734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":1590434734}]},"public_addr":"172.21.15.27:6805/1590434734","cluster_addr":"172.21.15.27:6806/1590434734","heartbeat_back_addr":"172.21.15.27:6808/1590434734","heartbeat_front_addr":"172.21.15.27:6807/1590434734","state":["exists","up"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":3010908863}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":3010908863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":3010908863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":3010908863}]},"public_addr":"172.21.15.27:6801/3010908863","cluster_addr":"172.21.15.27:6802/3010908863","heartbeat_back_addr":"172.21.15.27:6804/3010908863","heartbeat_front_addr":"172.21.15.27:6803/3010908863","state":["exists","up"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1661572621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1661572621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1661572621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1661572621}]},"public_addr":"172.21.15.27:6813/1661572621","cluster_addr":"172.21.15.27:6814/1661572621","heartbeat_back_addr":"172.21.15.27:6816/1661572621","heartbeat_front_addr":"172.21.15.27:6815/1661572621","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:27:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.418+0000 7f22257c2640 1 -- 172.21.15.27:0/4228896162 >> v1:172.21.15.27:6800/1098903051 conn(0x7f220c047fd0 legacy=0x7f220c04a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.418+0000 7f22257c2640 1 -- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22200904f0 msgr2=0x7f2220073100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.418+0000 7f22257c2640 1 --2- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22200904f0 0x7f2220073100 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f221000e9f0 tx=0x7f221000eeb0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.419+0000 7f22257c2640 1 -- 172.21.15.27:0/4228896162 shutdown_connections 2024-09-21T03:27:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.419+0000 7f22257c2640 1 --2- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f22200929f0 0x7f222006f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.419+0000 7f22257c2640 1 --2- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f22200920d0 0x7f2220073640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.419+0000 7f22257c2640 1 --2- 172.21.15.27:0/4228896162 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f22200904f0 0x7f2220073100 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.419+0000 7f22257c2640 1 -- 172.21.15.27:0/4228896162 >> 172.21.15.27:0/4228896162 conn(0x7f222009d760 msgr2=0x7f2220127550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.419+0000 7f22257c2640 1 -- 172.21.15.27:0/4228896162 shutdown_connections 2024-09-21T03:27:16.422 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.419+0000 7f22257c2640 1 -- 172.21.15.27:0/4228896162 wait complete. 2024-09-21T03:27:16.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T03:27:16.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T03:27:16.432 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T03:27:16.433 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T03:27:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f7c58da0640 1 -- 172.21.15.27:0/3231365544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5412c480 msgr2=0x7f7c54130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f7c58da0640 1 --2- 172.21.15.27:0/3231365544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5412c480 0x7f7c54130aa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7c48005fd0 tx=0x7f7c4802a4b0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f7c58da0640 1 -- 172.21.15.27:0/3231365544 shutdown_connections 2024-09-21T03:27:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f7c58da0640 1 --2- 172.21.15.27:0/3231365544 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7c54130fe0 0x7f7c541333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f7c58da0640 1 --2- 172.21.15.27:0/3231365544 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c5412c480 0x7f7c54130aa0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f7c58da0640 1 -- 172.21.15.27:0/3231365544 >> 172.21.15.27:0/3231365544 conn(0x7f7c54099cf0 msgr2=0x7f7c5409a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f7c58da0640 1 -- 172.21.15.27:0/3231365544 shutdown_connections 2024-09-21T03:27:16.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f7c58da0640 1 -- 172.21.15.27:0/3231365544 wait complete. 2024-09-21T03:27:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f5bbef25640 1 -- 172.21.15.27:0/3106165514 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5bb8093ae0 msgr2=0x7f5bb8093f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f5bbef25640 1 --2- 172.21.15.27:0/3106165514 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5bb8093ae0 0x7f5bb8093f30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0005fd0 tx=0x7f5bb002baa0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f5bbef25640 1 -- 172.21.15.27:0/3106165514 shutdown_connections 2024-09-21T03:27:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.542+0000 7f5bbef25640 1 --2- 172.21.15.27:0/3106165514 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5bb8093ae0 0x7f5bb8093f30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f5bbef25640 1 --2- 172.21.15.27:0/3106165514 >> v2:172.21.15.27:3300/0 conn(0x7f5bb812a650 0x7f5bb812aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f5bbef25640 1 -- 172.21.15.27:0/3106165514 >> 172.21.15.27:0/3106165514 conn(0x7f5bb80991f0 msgr2=0x7f5bb80968d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f5bbef25640 1 -- 172.21.15.27:0/3106165514 shutdown_connections 2024-09-21T03:27:16.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f5bbef25640 1 -- 172.21.15.27:0/3106165514 wait complete. 2024-09-21T03:27:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f7783ed7640 1 -- 172.21.15.27:0/2613001320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f777c072a60 msgr2=0x7f777c1288d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f7783ed7640 1 --2- 172.21.15.27:0/2613001320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f777c072a60 0x7f777c1288d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7774005fd0 tx=0x7f777402a730 comp rx=0 tx=0).stop 2024-09-21T03:27:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f7783ed7640 1 -- 172.21.15.27:0/2613001320 shutdown_connections 2024-09-21T03:27:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f7783ed7640 1 --2- 172.21.15.27:0/2613001320 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f777c072a60 0x7f777c1288d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f7783ed7640 1 --2- 172.21.15.27:0/2613001320 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f777c12bb80 0x7f777c128390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f7783ed7640 1 --2- 172.21.15.27:0/2613001320 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f777c12bfa0 0x7f777c12c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f7783ed7640 1 -- 172.21.15.27:0/2613001320 >> 172.21.15.27:0/2613001320 conn(0x7f777c09c350 msgr2=0x7f777c09c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7fca195bc640 1 -- 172.21.15.27:0/3316402332 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca14070240 msgr2=0x7fc9fc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7fca195bc640 1 --2- 172.21.15.27:0/3316402332 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca14070240 0x7fc9fc004800 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fca08006f70 tx=0x7fc9fc005830 comp rx=0 tx=0).stop 2024-09-21T03:27:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7fca195bc640 1 -- 172.21.15.27:0/3316402332 shutdown_connections 2024-09-21T03:27:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7fca195bc640 1 --2- 172.21.15.27:0/3316402332 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc9fc004d40 0x7fc9fc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7fca195bc640 1 --2- 172.21.15.27:0/3316402332 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca14070240 0x7fc9fc004800 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.548 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7fca195bc640 1 --2- 172.21.15.27:0/3316402332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9fc0039b0 0x7fc9fc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7fca195bc640 1 -- 172.21.15.27:0/3316402332 >> 172.21.15.27:0/3316402332 conn(0x7fca1409cbe0 msgr2=0x7fca1409cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f7783ed7640 1 -- 172.21.15.27:0/2613001320 shutdown_connections 2024-09-21T03:27:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7f7783ed7640 1 -- 172.21.15.27:0/2613001320 wait complete. 2024-09-21T03:27:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7fca195bc640 1 -- 172.21.15.27:0/3316402332 shutdown_connections 2024-09-21T03:27:16.549 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.543+0000 7fca195bc640 1 -- 172.21.15.27:0/3316402332 wait complete. 2024-09-21T03:27:16.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.544+0000 7f7c58da0640 1 Processor -- start 2024-09-21T03:27:16.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.544+0000 7f7c58da0640 1 -- start start 2024-09-21T03:27:16.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.544+0000 7f5bbef25640 1 Processor -- start 2024-09-21T03:27:16.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.544+0000 7f5bbef25640 1 -- start start 2024-09-21T03:27:16.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.544+0000 7f7783ed7640 1 Processor -- start 2024-09-21T03:27:16.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7783ed7640 1 -- start start 2024-09-21T03:27:16.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c58da0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c54094530 0x7f7c5406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.551 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c58da0640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7c54130fe0 0x7f7c54070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c58da0640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7c54071540 0x7f7c54070570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.552 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7fca195bc640 1 Processor -- start 2024-09-21T03:27:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c52ffd640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7c54130fe0 0x7f7c54070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c537fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c54094530 0x7f7c5406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c52ffd640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7c54130fe0 0x7f7c54070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:47072/0 (socket says 172.21.15.27:47072) 2024-09-21T03:27:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c537fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c54094530 0x7f7c5406faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40708/0 (socket says 172.21.15.27:40708) 2024-09-21T03:27:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c52ffd640 1 -- 172.21.15.27:0/464673983 learned_addr learned my addr 172.21.15.27:0/464673983 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7c541344d0 con 0x7f7c54094530 2024-09-21T03:27:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f7c54134350 con 0x7f7c54071540 2024-09-21T03:27:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f7c54134650 con 0x7f7c54130fe0 2024-09-21T03:27:16.553 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f5bbef25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bb8093ae0 0x7f5bb8130b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7783ed7640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f777c072a60 0x7f777c1c1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f5bbef25640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5bb8095cf0 0x7f5bb81310d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.554 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7783ed7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f777c12bb80 0x7f777c1c21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7781c73640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f777c072a60 0x7f777c1c1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7781c73640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f777c072a60 0x7f777c1c1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:56574/0 (socket says 172.21.15.27:56574) 2024-09-21T03:27:16.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7781c73640 1 -- 172.21.15.27:0/3829900682 learned_addr learned my addr 172.21.15.27:0/3829900682 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:16.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7783ed7640 1 --2- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f777c12bfa0 0x7f777c12f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f5bbef25640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5bb812a650 0x7f5bb8131610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.555 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7fca195bc640 1 -- start start 2024-09-21T03:27:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f5bb7fff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5bb8095cf0 0x7f5bb81310d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f5bb7fff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5bb8095cf0 0x7f5bb81310d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:47082/0 (socket says 172.21.15.27:47082) 2024-09-21T03:27:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f5bbd4c2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5bb812a650 0x7f5bb8131610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f5bbd4c2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5bb812a650 0x7f5bb8131610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:56586/0 (socket says 172.21.15.27:56586) 2024-09-21T03:27:16.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7782474640 1 --2- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f777c12bfa0 0x7f777c12f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f5bb7fff640 1 -- 172.21.15.27:0/1423762675 learned_addr learned my addr 172.21.15.27:0/1423762675 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb8132680 con 0x7f5bb8093ae0 2024-09-21T03:27:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb8132500 con 0x7f5bb812a650 2024-09-21T03:27:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.545+0000 7f7c53fff640 1 --2- 172.21.15.27:0/464673983 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7c54071540 0x7f7c54070570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f5bb8132800 con 0x7f5bb8095cf0 2024-09-21T03:27:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7c52ffd640 1 -- 172.21.15.27:0/464673983 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7c54071540 msgr2=0x7f7c54070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7c52ffd640 1 --2- 172.21.15.27:0/464673983 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7c54071540 0x7f7c54070570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7c52ffd640 1 -- 172.21.15.27:0/464673983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c54094530 msgr2=0x7f7c5406faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7c52ffd640 1 --2- 172.21.15.27:0/464673983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c54094530 0x7f7c5406faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f777c133590 con 0x7f777c12bb80 2024-09-21T03:27:16.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7c52ffd640 1 -- 172.21.15.27:0/464673983 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5412f920 con 0x7f7c54130fe0 2024-09-21T03:27:16.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7c53fff640 1 --2- 172.21.15.27:0/464673983 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7c54071540 0x7f7c54070570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:16.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7c52ffd640 1 --2- 172.21.15.27:0/464673983 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7c54130fe0 0x7f7c54070030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7c48006010 tx=0x7f7c480038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7fca195bc640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc9fc0039b0 0x7fca141c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7fca195bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9fc004d40 0x7fca141c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7fca12ffd640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc9fc0039b0 0x7fca141c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7fca12ffd640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc9fc0039b0 0x7fca141c7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:47106/0 (socket says 172.21.15.27:47106) 2024-09-21T03:27:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f777c133410 con 0x7f777c072a60 2024-09-21T03:27:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7781472640 1 --2- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f777c12bb80 0x7f777c1c21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f777c133710 con 0x7f777c12bfa0 2024-09-21T03:27:16.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f5bbccc1640 1 --2- 172.21.15.27:0/1423762675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bb8093ae0 0x7f5bb8130b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7c3bfff640 1 -- 172.21.15.27:0/464673983 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4803f070 con 0x7f7c54130fe0 2024-09-21T03:27:16.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.546+0000 7f7c3bfff640 1 -- 172.21.15.27:0/464673983 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7c480047a0 con 0x7f7c54130fe0 2024-09-21T03:27:16.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f7c3bfff640 1 -- 172.21.15.27:0/464673983 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48035030 con 0x7f7c54130fe0 2024-09-21T03:27:16.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7fca12ffd640 1 -- 172.21.15.27:0/92698997 learned_addr learned my addr 172.21.15.27:0/92698997 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:16.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7fca195bc640 1 --2- 172.21.15.27:0/92698997 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca14070240 0x7fca1412a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f5bb7fff640 1 -- 172.21.15.27:0/1423762675 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5bb812a650 msgr2=0x7f5bb8131610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f5bb7fff640 1 --2- 172.21.15.27:0/1423762675 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5bb812a650 0x7f5bb8131610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f5bb7fff640 1 -- 172.21.15.27:0/1423762675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bb8093ae0 msgr2=0x7f5bb8130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f5bb7fff640 1 --2- 172.21.15.27:0/1423762675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5bb8093ae0 0x7f5bb8130b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f7781c73640 1 -- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f777c12bfa0 msgr2=0x7f777c12f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f5bb7fff640 1 -- 172.21.15.27:0/1423762675 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb808c710 con 0x7f5bb8095cf0 2024-09-21T03:27:16.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7fca137fe640 1 --2- 172.21.15.27:0/92698997 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca14070240 0x7fca1412a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7fca127fc640 1 --2- 172.21.15.27:0/92698997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9fc004d40 0x7fca141c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fca140968a0 con 0x7fc9fc004d40 2024-09-21T03:27:16.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f7781c73640 1 --2- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f777c12bfa0 0x7f777c12f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fca14096720 con 0x7fca14070240 2024-09-21T03:27:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fca14096a20 con 0x7fc9fc0039b0 2024-09-21T03:27:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c54130210 con 0x7f7c54130fe0 2024-09-21T03:27:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c541c1920 con 0x7f7c54130fe0 2024-09-21T03:27:16.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f7781c73640 1 -- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f777c12bb80 msgr2=0x7f777c1c21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f5bbd4c2640 1 --2- 172.21.15.27:0/1423762675 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5bb812a650 0x7f5bb8131610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:16.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f5bb7fff640 1 --2- 172.21.15.27:0/1423762675 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5bb8095cf0 0x7f5bb81310d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4001870 tx=0x7f5ba4001d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:16.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7fca127fc640 1 -- 172.21.15.27:0/92698997 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc9fc0039b0 msgr2=0x7fca141c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f5bb57fa640 1 -- 172.21.15.27:0/1423762675 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba400a5c0 con 0x7f5bb8095cf0 2024-09-21T03:27:16.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f5bb57fa640 1 -- 172.21.15.27:0/1423762675 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5ba400a850 con 0x7f5bb8095cf0 2024-09-21T03:27:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f5bb57fa640 1 -- 172.21.15.27:0/1423762675 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba400aca0 con 0x7f5bb8095cf0 2024-09-21T03:27:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb808b350 con 0x7f5bb8095cf0 2024-09-21T03:27:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb8131d30 con 0x7f5bb8095cf0 2024-09-21T03:27:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7c5412a210 con 0x7f7c54130fe0 2024-09-21T03:27:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7fca127fc640 1 --2- 172.21.15.27:0/92698997 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc9fc0039b0 0x7fca141c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7fca127fc640 1 -- 172.21.15.27:0/92698997 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca14070240 msgr2=0x7fca1412a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7fca127fc640 1 --2- 172.21.15.27:0/92698997 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca14070240 0x7fca1412a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7fca127fc640 1 -- 172.21.15.27:0/92698997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca14090850 con 0x7fc9fc004d40 2024-09-21T03:27:16.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7fca137fe640 1 --2- 172.21.15.27:0/92698997 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca14070240 0x7fca1412a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7f7c3bfff640 1 -- 172.21.15.27:0/464673983 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f7c48042070 con 0x7f7c54130fe0 2024-09-21T03:27:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.549+0000 7f7c3bfff640 1 -- 172.21.15.27:0/464673983 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f7c4807e730 con 0x7f7c54130fe0 2024-09-21T03:27:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.549+0000 7f7c3bfff640 1 -- 172.21.15.27:0/464673983 --> v1:172.21.15.27:6801/3010908863 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c48035ec0 con 0x7f7c40052ab0 2024-09-21T03:27:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.549+0000 7f7c3bfff640 1 -- 172.21.15.27:0/464673983 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f7c48085050 con 0x7f7c54130fe0 2024-09-21T03:27:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f7781c73640 1 --2- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f777c12bb80 0x7f777c1c21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f7781c73640 1 -- 172.21.15.27:0/3829900682 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c1314d0 con 0x7f777c072a60 2024-09-21T03:27:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.547+0000 7f7781472640 1 --2- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f777c12bb80 0x7f777c1c21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7f7781c73640 1 --2- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f777c072a60 0x7f777c1c1c60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f77780092f0 tx=0x7f77780097b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:16.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7f7782474640 1 --2- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f777c12bfa0 0x7f777c12f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7f77727fc640 1 -- 172.21.15.27:0/3829900682 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777801f620 con 0x7f777c072a60 2024-09-21T03:27:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7f77727fc640 1 -- 172.21.15.27:0/3829900682 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f777801f8b0 con 0x7f777c072a60 2024-09-21T03:27:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7f77727fc640 1 -- 172.21.15.27:0/3829900682 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f777801fcc0 con 0x7f777c072a60 2024-09-21T03:27:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777c131030 con 0x7f777c072a60 2024-09-21T03:27:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.548+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f777c132220 con 0x7f777c072a60 2024-09-21T03:27:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.549+0000 7fca127fc640 1 --2- 172.21.15.27:0/92698997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9fc004d40 0x7fca141c8530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fca08030000 tx=0x7fca08001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.549+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5bb8062630 con 0x7f5bb8095cf0 2024-09-21T03:27:16.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.549+0000 7f5bb57fa640 1 -- 172.21.15.27:0/1423762675 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f5ba4021350 con 0x7f5bb8095cf0 2024-09-21T03:27:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.549+0000 7fc9fb7fe640 1 -- 172.21.15.27:0/92698997 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08031070 con 0x7fc9fc004d40 2024-09-21T03:27:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.550+0000 7fc9fb7fe640 1 -- 172.21.15.27:0/92698997 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca08006580 con 0x7fc9fc004d40 2024-09-21T03:27:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.550+0000 7fc9fb7fe640 1 -- 172.21.15.27:0/92698997 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08003a60 con 0x7fc9fc004d40 2024-09-21T03:27:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.550+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca1408e920 con 0x7fc9fc004d40 2024-09-21T03:27:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.550+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca141c8a70 con 0x7fc9fc004d40 2024-09-21T03:27:16.568 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.550+0000 7f77727fc640 1 -- 172.21.15.27:0/3829900682 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f7778027360 con 0x7f777c072a60 2024-09-21T03:27:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.550+0000 7f5bb57fa640 1 -- 172.21.15.27:0/1423762675 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f5ba405d550 con 0x7f5bb8095cf0 2024-09-21T03:27:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.550+0000 7f5bb57fa640 1 -- 172.21.15.27:0/1423762675 --> v1:172.21.15.27:6809/831015510 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b9c052340 con 0x7f5b9c04e650 2024-09-21T03:27:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.550+0000 7f5bb57fa640 1 -- 172.21.15.27:0/1423762675 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f5ba4064050 con 0x7f5bb8095cf0 2024-09-21T03:27:16.569 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.550+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f777c12a7f0 con 0x7f777c072a60 2024-09-21T03:27:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.551+0000 7f7c3bfff640 1 -- 172.21.15.27:0/464673983 <== osd.2 v1:172.21.15.27:6801/3010908863 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7c48035ec0 con 0x7f7c40052ab0 2024-09-21T03:27:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.551+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fca14096010 con 0x7fc9fc004d40 2024-09-21T03:27:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.551+0000 7f77727fc640 1 -- 172.21.15.27:0/3829900682 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f7778045390 con 0x7f777c072a60 2024-09-21T03:27:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.551+0000 7f77727fc640 1 -- 172.21.15.27:0/3829900682 --> v1:172.21.15.27:6813/1661572621 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7760052ac0 con 0x7f776004edb0 2024-09-21T03:27:16.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.551+0000 7f77727fc640 1 -- 172.21.15.27:0/3829900682 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f7778043d20 con 0x7f777c072a60 2024-09-21T03:27:16.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.551+0000 7fc9fb7fe640 1 -- 172.21.15.27:0/92698997 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7fca08034070 con 0x7fc9fc004d40 2024-09-21T03:27:16.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.552+0000 7fc9fb7fe640 1 -- 172.21.15.27:0/92698997 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7fca0807daa0 con 0x7fc9fc004d40 2024-09-21T03:27:16.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.552+0000 7fc9fb7fe640 1 -- 172.21.15.27:0/92698997 --> v1:172.21.15.27:6805/1590434734 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca08006290 con 0x7fca0004e650 2024-09-21T03:27:16.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.552+0000 7fc9fb7fe640 1 -- 172.21.15.27:0/92698997 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fca080822a0 con 0x7fc9fc004d40 2024-09-21T03:27:16.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.552+0000 7f5bb57fa640 1 -- 172.21.15.27:0/1423762675 <== osd.0 v1:172.21.15.27:6809/831015510 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5b9c052340 con 0x7f5b9c04e650 2024-09-21T03:27:16.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.552+0000 7f77727fc640 1 -- 172.21.15.27:0/3829900682 <== osd.3 v1:172.21.15.27:6813/1661572621 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7760052ac0 con 0x7f776004edb0 2024-09-21T03:27:16.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.553+0000 7fc9fb7fe640 1 -- 172.21.15.27:0/92698997 <== osd.1 v1:172.21.15.27:6805/1590434734 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fca08006290 con 0x7fca0004e650 2024-09-21T03:27:16.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 --> v1:172.21.15.27:6801/3010908863 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c5412bd70 con 0x7f7c40052ab0 2024-09-21T03:27:16.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f7c3bfff640 1 -- 172.21.15.27:0/464673983 <== osd.2 v1:172.21.15.27:6801/3010908863 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 1800444376) 0x7f7c5412bd70 con 0x7f7c40052ab0 2024-09-21T03:27:16.572 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 --> v1:172.21.15.27:6809/831015510 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5bb8062a60 con 0x7f5b9c04e650 2024-09-21T03:27:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 --> v1:172.21.15.27:6813/1661572621 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f777c12fa90 con 0x7f776004edb0 2024-09-21T03:27:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 >> v1:172.21.15.27:6801/3010908863 conn(0x7f7c40052ab0 legacy=0x7f7c40054f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.573 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f5bb57fa640 1 -- 172.21.15.27:0/1423762675 <== osd.0 v1:172.21.15.27:6809/831015510 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 654699149) 0x7f5bb8062a60 con 0x7f5b9c04e650 2024-09-21T03:27:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 >> v1:172.21.15.27:6800/1098903051 conn(0x7f7c4004c430 legacy=0x7f7c4004e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7c54130fe0 msgr2=0x7f7c54070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f7c58da0640 1 --2- 172.21.15.27:0/464673983 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7c54130fe0 0x7f7c54070030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7c48006010 tx=0x7f7c480038a0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f7c53fff640 1 -- 172.21.15.27:0/464673983 reap_dead start 2024-09-21T03:27:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 shutdown_connections 2024-09-21T03:27:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.566+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 >> 172.21.15.27:0/464673983 conn(0x7f7c54099cf0 msgr2=0x7f7c5412a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.574 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f77727fc640 1 -- 172.21.15.27:0/3829900682 <== osd.3 v1:172.21.15.27:6813/1661572621 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 1800444376) 0x7f777c12fa90 con 0x7f776004edb0 2024-09-21T03:27:16.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 shutdown_connections 2024-09-21T03:27:16.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f7c58da0640 1 -- 172.21.15.27:0/464673983 wait complete. 2024-09-21T03:27:16.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 >> v1:172.21.15.27:6809/831015510 conn(0x7f5b9c04e650 legacy=0x7f5b9c050ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 >> v1:172.21.15.27:6813/1661572621 conn(0x7f776004edb0 legacy=0x7f7760051250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 >> v1:172.21.15.27:6800/1098903051 conn(0x7f5b9c047fd0 legacy=0x7f5b9c04a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5bb8095cf0 msgr2=0x7f5bb81310d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f5bbef25640 1 --2- 172.21.15.27:0/1423762675 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5bb8095cf0 0x7f5bb81310d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4001870 tx=0x7f5ba4001d30 comp rx=0 tx=0).stop 2024-09-21T03:27:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f5bbd4c2640 1 -- 172.21.15.27:0/1423762675 reap_dead start 2024-09-21T03:27:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 shutdown_connections 2024-09-21T03:27:16.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 >> 172.21.15.27:0/1423762675 conn(0x7f5bb80991f0 msgr2=0x7f5bb809a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 >> v1:172.21.15.27:6800/1098903051 conn(0x7f7760048020 legacy=0x7f776004a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f777c072a60 msgr2=0x7f777c1c1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f7783ed7640 1 --2- 172.21.15.27:0/3829900682 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f777c072a60 0x7f777c1c1c60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f77780092f0 tx=0x7f77780097b0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f7782474640 1 -- 172.21.15.27:0/3829900682 reap_dead start 2024-09-21T03:27:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 shutdown_connections 2024-09-21T03:27:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.567+0000 7f5bbef25640 1 -- 172.21.15.27:0/1423762675 wait complete. 2024-09-21T03:27:16.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.568+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 shutdown_connections 2024-09-21T03:27:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.568+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 >> 172.21.15.27:0/3829900682 conn(0x7f777c09c350 msgr2=0x7f777c12afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.568+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 --> v1:172.21.15.27:6805/1590434734 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fca14091740 con 0x7fca0004e650 2024-09-21T03:27:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.568+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 shutdown_connections 2024-09-21T03:27:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.568+0000 7f7783ed7640 1 -- 172.21.15.27:0/3829900682 wait complete. 2024-09-21T03:27:16.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.568+0000 7fc9fb7fe640 1 -- 172.21.15.27:0/92698997 <== osd.1 v1:172.21.15.27:6805/1590434734 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3659784045) 0x7fca14091740 con 0x7fca0004e650 2024-09-21T03:27:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.568+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 >> v1:172.21.15.27:6805/1590434734 conn(0x7fca0004e650 legacy=0x7fca00050ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.568+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 >> v1:172.21.15.27:6800/1098903051 conn(0x7fca00047fd0 legacy=0x7fca0004a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.568+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9fc004d40 msgr2=0x7fca141c8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.568+0000 7fca195bc640 1 --2- 172.21.15.27:0/92698997 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9fc004d40 0x7fca141c8530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fca08030000 tx=0x7fca08001c10 comp rx=0 tx=0).stop 2024-09-21T03:27:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.568+0000 7fca137fe640 1 -- 172.21.15.27:0/92698997 reap_dead start 2024-09-21T03:27:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.569+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 shutdown_connections 2024-09-21T03:27:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.569+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 >> 172.21.15.27:0/92698997 conn(0x7fca1409cbe0 msgr2=0x7fca14091030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.569+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 shutdown_connections 2024-09-21T03:27:16.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.569+0000 7fca195bc640 1 -- 172.21.15.27:0/92698997 wait complete. 2024-09-21T03:27:16.583 INFO:teuthology.orchestra.run.smithi027.stdout:47244640259 2024-09-21T03:27:16.583 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T03:27:16.584 INFO:teuthology.orchestra.run.smithi027.stdout:47244640259 2024-09-21T03:27:16.584 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T03:27:16.585 INFO:teuthology.orchestra.run.smithi027.stdout:47244640260 2024-09-21T03:27:16.585 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T03:27:16.586 INFO:teuthology.orchestra.run.smithi027.stdout:38654705668 2024-09-21T03:27:16.586 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T03:27:16.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/3978003719 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa9e412fab0 msgr2=0x7fa9e4131ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f80442b7640 1 -- 172.21.15.27:0/3110088539 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f803c094a20 msgr2=0x7f803c094e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.735 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f80442b7640 1 --2- 172.21.15.27:0/3110088539 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f803c094a20 0x7f803c094e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8038009930 tx=0x7f803802a510 comp rx=0 tx=0).stop 2024-09-21T03:27:16.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7fa9eb4d0640 1 --2- 172.21.15.27:0/3978003719 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa9e412fab0 0x7fa9e4131ea0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc009be0 tx=0x7fa9dc02b980 comp rx=0 tx=0).stop 2024-09-21T03:27:16.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f9152c0d640 1 -- 172.21.15.27:0/3351709639 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f914c094a20 msgr2=0x7f914c094e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f9152c0d640 1 --2- 172.21.15.27:0/3351709639 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f914c094a20 0x7f914c094e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f914000d910 tx=0x7f914002c4b0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f9152c0d640 1 -- 172.21.15.27:0/3351709639 shutdown_connections 2024-09-21T03:27:16.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f9152c0d640 1 --2- 172.21.15.27:0/3351709639 >> v2:172.21.15.27:3300/0 conn(0x7f914c12b090 0x7f914c12b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.736 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f9152c0d640 1 --2- 172.21.15.27:0/3351709639 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f914c094a20 0x7f914c094e00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f9152c0d640 1 -- 172.21.15.27:0/3351709639 >> 172.21.15.27:0/3351709639 conn(0x7f914c09b0a0 msgr2=0x7f914c09b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f80442b7640 1 -- 172.21.15.27:0/3110088539 shutdown_connections 2024-09-21T03:27:16.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f80442b7640 1 --2- 172.21.15.27:0/3110088539 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f803c12b090 0x7f803c12b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f80442b7640 1 --2- 172.21.15.27:0/3110088539 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f803c094a20 0x7f803c094e00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.737 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f80442b7640 1 -- 172.21.15.27:0/3110088539 >> 172.21.15.27:0/3110088539 conn(0x7f803c09b0a0 msgr2=0x7f803c09b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f9152c0d640 1 -- 172.21.15.27:0/3351709639 shutdown_connections 2024-09-21T03:27:16.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f80442b7640 1 -- 172.21.15.27:0/3110088539 shutdown_connections 2024-09-21T03:27:16.738 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f9152c0d640 1 -- 172.21.15.27:0/3351709639 wait complete. 2024-09-21T03:27:16.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/3978003719 shutdown_connections 2024-09-21T03:27:16.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f00221a5640 1 -- 172.21.15.27:0/1667144593 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f001c12bfa0 msgr2=0x7f001c12c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f00221a5640 1 --2- 172.21.15.27:0/1667144593 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f001c12bfa0 0x7f001c12c380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f000c005fd0 tx=0x7f000c02a4b0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f80442b7640 1 -- 172.21.15.27:0/3110088539 wait complete. 2024-09-21T03:27:16.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7fa9eb4d0640 1 --2- 172.21.15.27:0/3978003719 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa9e412fab0 0x7fa9e4131ea0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7fa9eb4d0640 1 --2- 172.21.15.27:0/3978003719 >> v2:172.21.15.27:3300/0 conn(0x7fa9e4070250 0x7fa9e412f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/3978003719 >> 172.21.15.27:0/3978003719 conn(0x7fa9e4099510 msgr2=0x7fa9e4099920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f00221a5640 1 -- 172.21.15.27:0/1667144593 shutdown_connections 2024-09-21T03:27:16.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f00221a5640 1 --2- 172.21.15.27:0/1667144593 >> v2:172.21.15.27:3300/0 conn(0x7f001c12bb80 0x7f001c128390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f00221a5640 1 --2- 172.21.15.27:0/1667144593 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f001c12bfa0 0x7f001c12c380 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f00221a5640 1 -- 172.21.15.27:0/1667144593 >> 172.21.15.27:0/1667144593 conn(0x7f001c09c350 msgr2=0x7f001c09c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f00221a5640 1 -- 172.21.15.27:0/1667144593 shutdown_connections 2024-09-21T03:27:16.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7f00221a5640 1 -- 172.21.15.27:0/1667144593 wait complete. 2024-09-21T03:27:16.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/3978003719 shutdown_connections 2024-09-21T03:27:16.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.733+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/3978003719 wait complete. 2024-09-21T03:27:16.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.734+0000 7f9152c0d640 1 Processor -- start 2024-09-21T03:27:16.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.734+0000 7f80442b7640 1 Processor -- start 2024-09-21T03:27:16.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.734+0000 7f00221a5640 1 Processor -- start 2024-09-21T03:27:16.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.734+0000 7f9152c0d640 1 -- start start 2024-09-21T03:27:16.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.734+0000 7f80442b7640 1 -- start start 2024-09-21T03:27:16.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f00221a5640 1 -- start start 2024-09-21T03:27:16.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7fa9eb4d0640 1 Processor -- start 2024-09-21T03:27:16.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7fa9eb4d0640 1 -- start start 2024-09-21T03:27:16.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f00221a5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001c072a60 0x7f001c1c1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f00221a5640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f001c12bb80 0x7f001c1c2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f80442b7640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f803c094a20 0x7f803c1c0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f00221a5640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f001c1c25a0 0x7f001c12f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f001b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001c072a60 0x7f001c1c1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f001affd640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f001c12bb80 0x7f001c1c2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f001affd640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f001c12bb80 0x7f001c1c2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:56624/0 (socket says 172.21.15.27:56624) 2024-09-21T03:27:16.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f9152c0d640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f914c094a20 0x7f914c093250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f80442b7640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f803c095340 0x7f803c1c12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f80442b7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c12b090 0x7f803c0718a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f80442b7640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f803c1336a0 con 0x7f803c12b090 2024-09-21T03:27:16.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f80442b7640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f803c133520 con 0x7f803c095340 2024-09-21T03:27:16.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f80442b7640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f803c133820 con 0x7f803c094a20 2024-09-21T03:27:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f001b7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001c072a60 0x7f001c1c1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40786/0 (socket says 172.21.15.27:40786) 2024-09-21T03:27:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f001affd640 1 -- 172.21.15.27:0/156028805 learned_addr learned my addr 172.21.15.27:0/156028805 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f001c1346d0 con 0x7f001c072a60 2024-09-21T03:27:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f8042854640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c12b090 0x7f803c0718a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f8042053640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f803c094a20 0x7f803c1c0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f8042854640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c12b090 0x7f803c0718a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40802/0 (socket says 172.21.15.27:40802) 2024-09-21T03:27:16.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f8042854640 1 -- 172.21.15.27:0/2217369703 learned_addr learned my addr 172.21.15.27:0/2217369703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:16.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f8042053640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f803c094a20 0x7f803c1c0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:47152/0 (socket says 172.21.15.27:47152) 2024-09-21T03:27:16.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f001bfff640 1 --2- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f001c1c25a0 0x7f001c12f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f001c134550 con 0x7f001c12bb80 2024-09-21T03:27:16.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f001c134850 con 0x7f001c1c25a0 2024-09-21T03:27:16.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.735+0000 7f8041852640 1 --2- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f803c095340 0x7f803c1c12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f9152c0d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f914c095340 0x7f914c093790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f001affd640 1 -- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f001c1c25a0 msgr2=0x7f001c12f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f001affd640 1 --2- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f001c1c25a0 0x7f001c12f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f001affd640 1 -- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001c072a60 msgr2=0x7f001c1c1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f001affd640 1 --2- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001c072a60 0x7f001c1c1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f001affd640 1 -- 172.21.15.27:0/156028805 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f001c131c30 con 0x7f001c12bb80 2024-09-21T03:27:16.748 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f001bfff640 1 --2- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f001c1c25a0 0x7f001c12f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:27:16.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f8042053640 1 -- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f803c095340 msgr2=0x7f803c1c12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f8042053640 1 --2- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f803c095340 0x7f803c1c12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f8042053640 1 -- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c12b090 msgr2=0x7f803c0718a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f8042053640 1 --2- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c12b090 0x7f803c0718a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f8042053640 1 -- 172.21.15.27:0/2217369703 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f803c0733d0 con 0x7f803c094a20 2024-09-21T03:27:16.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f91509a9640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f914c094a20 0x7f914c093250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f001affd640 1 --2- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f001c12bb80 0x7f001c1c2060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f001000da70 tx=0x7f001000df30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:16.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f91509a9640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f914c094a20 0x7f914c093250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:47164/0 (socket says 172.21.15.27:47164) 2024-09-21T03:27:16.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f914bfff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f914c095340 0x7f914c093790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f914bfff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f914c095340 0x7f914c093790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:56648/0 (socket says 172.21.15.27:56648) 2024-09-21T03:27:16.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f91509a9640 1 -- 172.21.15.27:0/219728371 learned_addr learned my addr 172.21.15.27:0/219728371 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:16.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f8042053640 1 --2- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f803c094a20 0x7f803c1c0d90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8038031000 tx=0x7f80380038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:16.750 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f9152c0d640 1 --2- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f914c12b090 0x7f914c071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f8042854640 1 --2- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c12b090 0x7f803c0718a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:16.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f91511aa640 1 --2- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f914c12b090 0x7f914c071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f8041852640 1 --2- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f803c095340 0x7f803c1c12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:16.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f914c06ddf0 con 0x7f914c12b090 2024-09-21T03:27:16.751 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f914c06dc70 con 0x7f914c095340 2024-09-21T03:27:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f0003fff640 1 -- 172.21.15.27:0/156028805 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0010017030 con 0x7f001c12bb80 2024-09-21T03:27:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f0003fff640 1 -- 172.21.15.27:0/156028805 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0010004110 con 0x7f001c12bb80 2024-09-21T03:27:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f0003fff640 1 -- 172.21.15.27:0/156028805 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00100099a0 con 0x7f001c12bb80 2024-09-21T03:27:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f8032ffd640 1 -- 172.21.15.27:0/2217369703 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803803f070 con 0x7f803c094a20 2024-09-21T03:27:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f914c06df70 con 0x7f914c094a20 2024-09-21T03:27:16.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.736+0000 7f80442b7640 1 -- 172.21.15.27:0/2217369703 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f803c12bd50 con 0x7f803c094a20 2024-09-21T03:27:16.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7f8032ffd640 1 -- 172.21.15.27:0/2217369703 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f80380047a0 con 0x7f803c094a20 2024-09-21T03:27:16.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7f8032ffd640 1 -- 172.21.15.27:0/2217369703 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8038035030 con 0x7f803c094a20 2024-09-21T03:27:16.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7fa9eb4d0640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa9e4070250 0x7fa9e4096280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7f80442b7640 1 -- 172.21.15.27:0/2217369703 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f803c1c1810 con 0x7f803c094a20 2024-09-21T03:27:16.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7fa9eb4d0640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9e412a210 0x7fa9e40967c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f001c131030 con 0x7f001c12bb80 2024-09-21T03:27:16.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9e8a6b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9e412a210 0x7fa9e40967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7f91511aa640 1 -- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f914c094a20 msgr2=0x7f914c093250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7f91511aa640 1 --2- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f914c094a20 0x7f914c093250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7f91511aa640 1 -- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f914c095340 msgr2=0x7f914c093790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7f91511aa640 1 --2- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f914c095340 0x7f914c093790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7f91511aa640 1 -- 172.21.15.27:0/219728371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914c073590 con 0x7f914c12b090 2024-09-21T03:27:16.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7f91511aa640 1 --2- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f914c12b090 0x7f914c071aa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f914400da40 tx=0x7f914400df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:16.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.737+0000 7fa9e926c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa9e4070250 0x7fa9e4096280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:16.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9e926c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa9e4070250 0x7fa9e4096280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:56664/0 (socket says 172.21.15.27:56664) 2024-09-21T03:27:16.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9eb4d0640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa9e412fab0 0x7fa9e412b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:16.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9eb4d0640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e4132770 con 0x7fa9e412a210 2024-09-21T03:27:16.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9e926c640 1 -- 172.21.15.27:0/1547845130 learned_addr learned my addr 172.21.15.27:0/1547845130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:16.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e41325f0 con 0x7fa9e4070250 2024-09-21T03:27:16.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fa9e41328f0 con 0x7fa9e412fab0 2024-09-21T03:27:16.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f001c12fa00 con 0x7f001c12bb80 2024-09-21T03:27:16.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7f80442b7640 1 -- 172.21.15.27:0/2217369703 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f803c094e00 con 0x7f803c094a20 2024-09-21T03:27:16.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7f91497fa640 1 -- 172.21.15.27:0/219728371 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914401c070 con 0x7f914c12b090 2024-09-21T03:27:16.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9e926c640 1 -- 172.21.15.27:0/1547845130 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa9e412fab0 msgr2=0x7fa9e412b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9e926c640 1 --2- 172.21.15.27:0/1547845130 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa9e412fab0 0x7fa9e412b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9e926c640 1 -- 172.21.15.27:0/1547845130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9e412a210 msgr2=0x7fa9e40967c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9e926c640 1 --2- 172.21.15.27:0/1547845130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9e412a210 0x7fa9e40967c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7fa9e926c640 1 -- 172.21.15.27:0/1547845130 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e412cfb0 con 0x7fa9e4070250 2024-09-21T03:27:16.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7f91497fa640 1 -- 172.21.15.27:0/219728371 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f914401a070 con 0x7f914c12b090 2024-09-21T03:27:16.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7f91497fa640 1 -- 172.21.15.27:0/219728371 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144016e30 con 0x7f914c12b090 2024-09-21T03:27:16.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f914c093cd0 con 0x7f914c12b090 2024-09-21T03:27:16.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.739+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f914c12c580 con 0x7f914c12b090 2024-09-21T03:27:16.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7f8032ffd640 1 -- 172.21.15.27:0/2217369703 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f8038042070 con 0x7f803c094a20 2024-09-21T03:27:16.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.739+0000 7fa9e926c640 1 --2- 172.21.15.27:0/1547845130 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa9e4070250 0x7fa9e4096280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000c410 tx=0x7fa9e000c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:16.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.739+0000 7f8032ffd640 1 -- 172.21.15.27:0/2217369703 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f803807eab0 con 0x7f803c094a20 2024-09-21T03:27:16.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.739+0000 7fa9d9ffb640 1 -- 172.21.15.27:0/1547845130 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0017070 con 0x7fa9e4070250 2024-09-21T03:27:16.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.739+0000 7fa9d9ffb640 1 -- 172.21.15.27:0/1547845130 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa9e00035f0 con 0x7fa9e4070250 2024-09-21T03:27:16.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.739+0000 7fa9d9ffb640 1 -- 172.21.15.27:0/1547845130 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0003a40 con 0x7fa9e4070250 2024-09-21T03:27:16.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.738+0000 7f0003fff640 1 -- 172.21.15.27:0/156028805 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f0010009b40 con 0x7f001c12bb80 2024-09-21T03:27:16.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.739+0000 7f0003fff640 1 -- 172.21.15.27:0/156028805 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f001003c490 con 0x7f001c12bb80 2024-09-21T03:27:16.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.740+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e412e2d0 con 0x7fa9e4070250 2024-09-21T03:27:16.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.740+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f001c12c380 con 0x7f001c12bb80 2024-09-21T03:27:16.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.740+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e412bca0 con 0x7fa9e4070250 2024-09-21T03:27:16.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.740+0000 7f91497fa640 1 -- 172.21.15.27:0/219728371 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f914400b900 con 0x7f914c12b090 2024-09-21T03:27:16.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.741+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f914c12b4a0 con 0x7f914c12b090 2024-09-21T03:27:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.741+0000 7f91497fa640 1 -- 172.21.15.27:0/219728371 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f9144066250 con 0x7f914c12b090 2024-09-21T03:27:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.741+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e412a800 con 0x7fa9e4070250 2024-09-21T03:27:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.744+0000 7f8032ffd640 1 -- 172.21.15.27:0/2217369703 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f803807dc00 con 0x7f803c094a20 2024-09-21T03:27:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.745+0000 7f0003fff640 1 -- 172.21.15.27:0/156028805 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f001c131c30 con 0x7f001c12bb80 2024-09-21T03:27:16.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.745+0000 7f91497fa640 1 -- 172.21.15.27:0/219728371 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f914c12b4a0 con 0x7f914c12b090 2024-09-21T03:27:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.745+0000 7fa9d9ffb640 1 -- 172.21.15.27:0/1547845130 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7fa9e0003be0 con 0x7fa9e4070250 2024-09-21T03:27:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.746+0000 7fa9d9ffb640 1 -- 172.21.15.27:0/1547845130 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7fa9e005cfc0 con 0x7fa9e4070250 2024-09-21T03:27:16.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.749+0000 7fa9d9ffb640 1 -- 172.21.15.27:0/1547845130 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7fa9e003c230 con 0x7fa9e4070250 2024-09-21T03:27:16.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.895+0000 7f80442b7640 1 -- 172.21.15.27:0/2217369703 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f803c067d30 con 0x7f803c094a20 2024-09-21T03:27:16.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.896+0000 7f8032ffd640 1 -- 172.21.15.27:0/2217369703 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f803807d740 con 0x7f803c094a20 2024-09-21T03:27:16.898 INFO:teuthology.orchestra.run.smithi027.stdout:47244640259 2024-09-21T03:27:16.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.897+0000 7f80442b7640 1 -- 172.21.15.27:0/2217369703 >> v1:172.21.15.27:6800/1098903051 conn(0x7f8020047fd0 legacy=0x7f802004a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.897+0000 7f80442b7640 1 -- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f803c094a20 msgr2=0x7f803c1c0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.897+0000 7f80442b7640 1 --2- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f803c094a20 0x7f803c1c0d90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8038031000 tx=0x7f80380038a0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.898+0000 7f80442b7640 1 -- 172.21.15.27:0/2217369703 shutdown_connections 2024-09-21T03:27:16.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.898+0000 7f80442b7640 1 --2- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c12b090 0x7f803c0718a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.898+0000 7f80442b7640 1 --2- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f803c095340 0x7f803c1c12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.898+0000 7f80442b7640 1 --2- 172.21.15.27:0/2217369703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f803c094a20 0x7f803c1c0d90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.898+0000 7f80442b7640 1 -- 172.21.15.27:0/2217369703 >> 172.21.15.27:0/2217369703 conn(0x7f803c09b0a0 msgr2=0x7f803c1274e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.898+0000 7f80442b7640 1 -- 172.21.15.27:0/2217369703 shutdown_connections 2024-09-21T03:27:16.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.898+0000 7f80442b7640 1 -- 172.21.15.27:0/2217369703 wait complete. 2024-09-21T03:27:16.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.902+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa9e412ed30 con 0x7fa9e4070250 2024-09-21T03:27:16.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.903+0000 7fa9d9ffb640 1 -- 172.21.15.27:0/1547845130 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa9e003c490 con 0x7fa9e4070250 2024-09-21T03:27:16.905 INFO:teuthology.orchestra.run.smithi027.stdout:47244640258 2024-09-21T03:27:16.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.905+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 >> v1:172.21.15.27:6800/1098903051 conn(0x7fa9c804c3e0 legacy=0x7fa9c804e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.905+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa9e4070250 msgr2=0x7fa9e4096280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.907 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.905+0000 7fa9eb4d0640 1 --2- 172.21.15.27:0/1547845130 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa9e4070250 0x7fa9e4096280 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000c410 tx=0x7fa9e000c810 comp rx=0 tx=0).stop 2024-09-21T03:27:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.905+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 shutdown_connections 2024-09-21T03:27:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.905+0000 7fa9eb4d0640 1 --2- 172.21.15.27:0/1547845130 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa9e412fab0 0x7fa9e412b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.905+0000 7fa9eb4d0640 1 --2- 172.21.15.27:0/1547845130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9e412a210 0x7fa9e40967c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.905+0000 7fa9eb4d0640 1 --2- 172.21.15.27:0/1547845130 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa9e4070250 0x7fa9e4096280 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.905+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 >> 172.21.15.27:0/1547845130 conn(0x7fa9e4099510 msgr2=0x7fa9e4097230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.905+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 shutdown_connections 2024-09-21T03:27:16.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.905+0000 7fa9eb4d0640 1 -- 172.21.15.27:0/1547845130 wait complete. 2024-09-21T03:27:16.915 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.0 2024-09-21T03:27:16.916 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.914+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f001c067d30 con 0x7f001c12bb80 2024-09-21T03:27:16.917 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.915+0000 7f0003fff640 1 -- 172.21.15.27:0/156028805 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f001003c6f0 con 0x7f001c12bb80 2024-09-21T03:27:16.917 INFO:teuthology.orchestra.run.smithi027.stdout:47244640258 2024-09-21T03:27:16.918 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.2 2024-09-21T03:27:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.916+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 >> v1:172.21.15.27:6800/1098903051 conn(0x7f0008048020 legacy=0x7f000804a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.916+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f001c12bb80 msgr2=0x7f001c1c2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.916+0000 7f00221a5640 1 --2- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f001c12bb80 0x7f001c1c2060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f001000da70 tx=0x7f001000df30 comp rx=0 tx=0).stop 2024-09-21T03:27:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.916+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 shutdown_connections 2024-09-21T03:27:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.916+0000 7f00221a5640 1 --2- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f001c1c25a0 0x7f001c12f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.916+0000 7f00221a5640 1 --2- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f001c12bb80 0x7f001c1c2060 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.916+0000 7f00221a5640 1 --2- 172.21.15.27:0/156028805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f001c072a60 0x7f001c1c1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.919 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.916+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 >> 172.21.15.27:0/156028805 conn(0x7f001c09c350 msgr2=0x7f001c12af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.917+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 shutdown_connections 2024-09-21T03:27:16.920 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.917+0000 7f00221a5640 1 -- 172.21.15.27:0/156028805 wait complete. 2024-09-21T03:27:16.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.922+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f914c129f80 con 0x7f914c12b090 2024-09-21T03:27:16.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.922+0000 7f91497fa640 1 -- 172.21.15.27:0/219728371 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f914c129f80 con 0x7f914c12b090 2024-09-21T03:27:16.925 INFO:teuthology.orchestra.run.smithi027.stdout:38654705667 2024-09-21T03:27:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.924+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 >> v1:172.21.15.27:6800/1098903051 conn(0x7f9130047fd0 legacy=0x7f913004a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.924+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f914c12b090 msgr2=0x7f914c071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.924+0000 7f9152c0d640 1 --2- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f914c12b090 0x7f914c071aa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f914400da40 tx=0x7f914400df00 comp rx=0 tx=0).stop 2024-09-21T03:27:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.924+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 shutdown_connections 2024-09-21T03:27:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.924+0000 7f9152c0d640 1 --2- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f914c12b090 0x7f914c071aa0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.924+0000 7f9152c0d640 1 --2- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f914c095340 0x7f914c093790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.924+0000 7f9152c0d640 1 --2- 172.21.15.27:0/219728371 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f914c094a20 0x7f914c093250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.924+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 >> 172.21.15.27:0/219728371 conn(0x7f914c09b0a0 msgr2=0x7f914c098ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.924+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 shutdown_connections 2024-09-21T03:27:16.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:16.924+0000 7f9152c0d640 1 -- 172.21.15.27:0/219728371 wait complete. 2024-09-21T03:27:16.930 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-09-21T03:27:16.937 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.1 2024-09-21T03:27:17.917 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T03:27:17.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T03:27:17.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T03:27:17.938 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T03:27:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7f558fabe640 1 -- 172.21.15.27:0/2544323053 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f558812e340 msgr2=0x7f558812e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7f558fabe640 1 --2- 172.21.15.27:0/2544323053 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f558812e340 0x7f558812e790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5580005fd0 tx=0x7f558002a980 comp rx=0 tx=0).stop 2024-09-21T03:27:18.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7f558fabe640 1 -- 172.21.15.27:0/2544323053 shutdown_connections 2024-09-21T03:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7f558fabe640 1 --2- 172.21.15.27:0/2544323053 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f558812e340 0x7f558812e790 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7f558fabe640 1 --2- 172.21.15.27:0/2544323053 >> v2:172.21.15.27:3300/0 conn(0x7f5588097f10 0x7f55880982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7f558fabe640 1 -- 172.21.15.27:0/2544323053 >> 172.21.15.27:0/2544323053 conn(0x7f558809e590 msgr2=0x7f558809e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7fa2bb92c640 1 -- 172.21.15.27:0/830045748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b412bfa0 msgr2=0x7fa2b412c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7fa2bb92c640 1 --2- 172.21.15.27:0/830045748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b412bfa0 0x7fa2b412c380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0005fd0 tx=0x7fa2b002a4b0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7f558fabe640 1 -- 172.21.15.27:0/2544323053 shutdown_connections 2024-09-21T03:27:18.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7f558fabe640 1 -- 172.21.15.27:0/2544323053 wait complete. 2024-09-21T03:27:18.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7fa2bb92c640 1 -- 172.21.15.27:0/830045748 shutdown_connections 2024-09-21T03:27:18.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7fa2bb92c640 1 --2- 172.21.15.27:0/830045748 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa2b4072a60 0x7fa2b4128aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7fa2bb92c640 1 --2- 172.21.15.27:0/830045748 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b412bfa0 0x7fa2b412c380 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.042 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7fa2bb92c640 1 -- 172.21.15.27:0/830045748 >> 172.21.15.27:0/830045748 conn(0x7fa2b409c350 msgr2=0x7fa2b409c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:18.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7fa2bb92c640 1 -- 172.21.15.27:0/830045748 shutdown_connections 2024-09-21T03:27:18.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.038+0000 7fa2bb92c640 1 -- 172.21.15.27:0/830045748 wait complete. 2024-09-21T03:27:18.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.039+0000 7f558fabe640 1 Processor -- start 2024-09-21T03:27:18.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.039+0000 7f558fabe640 1 -- start start 2024-09-21T03:27:18.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.039+0000 7fa2bb92c640 1 Processor -- start 2024-09-21T03:27:18.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.039+0000 7fa2bb92c640 1 -- start start 2024-09-21T03:27:18.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.039+0000 7f558fabe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5588097f10 0x7f5588070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558fabe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5588098830 0x7f5588070850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558fabe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f558812e340 0x7f5588070d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558fabe640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5588136960 con 0x7f5588098830 2024-09-21T03:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558fabe640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f55881367e0 con 0x7f558812e340 2024-09-21T03:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558d85a640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5588097f10 0x7f5588070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558fabe640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f5588136ae0 con 0x7f5588097f10 2024-09-21T03:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558d85a640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5588097f10 0x7f5588070310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:47210/0 (socket says 172.21.15.27:47210) 2024-09-21T03:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558e05b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f558812e340 0x7f5588070d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558d85a640 1 -- 172.21.15.27:0/4120284672 learned_addr learned my addr 172.21.15.27:0/4120284672 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:18.045 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558d059640 1 --2- 172.21.15.27:0/4120284672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5588098830 0x7f5588070850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7fa2bb92c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa2b4072a60 0x7fa2b41239b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7fa2bb92c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa2b412bb80 0x7fa2b4123ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7fa2bb92c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b412bfa0 0x7fa2b4124430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7fa2bb92c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b4134690 con 0x7fa2b412bfa0 2024-09-21T03:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7fa2bb92c640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b4134510 con 0x7fa2b412bb80 2024-09-21T03:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7fa2bb92c640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fa2b4134810 con 0x7fa2b4072a60 2024-09-21T03:27:18.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7fa2b8ec7640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa2b412bb80 0x7fa2b4123ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7fa2b8ec7640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa2b412bb80 0x7fa2b4123ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:56698/0 (socket says 172.21.15.27:56698) 2024-09-21T03:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7fa2b8ec7640 1 -- 172.21.15.27:0/2310973194 learned_addr learned my addr 172.21.15.27:0/2310973194 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7fa2b96c8640 1 --2- 172.21.15.27:0/2310973194 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa2b4072a60 0x7fa2b41239b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558e05b640 1 -- 172.21.15.27:0/4120284672 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5588097f10 msgr2=0x7f5588070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558e05b640 1 --2- 172.21.15.27:0/4120284672 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5588097f10 0x7f5588070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558e05b640 1 -- 172.21.15.27:0/4120284672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5588098830 msgr2=0x7f5588070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558e05b640 1 --2- 172.21.15.27:0/4120284672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5588098830 0x7f5588070850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.040+0000 7f558e05b640 1 -- 172.21.15.27:0/4120284672 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55880726d0 con 0x7f558812e340 2024-09-21T03:27:18.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7fa2b8ec7640 1 -- 172.21.15.27:0/2310973194 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa2b4072a60 msgr2=0x7fa2b41239b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7fa2b8ec7640 1 --2- 172.21.15.27:0/2310973194 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa2b4072a60 0x7fa2b41239b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7fa2b8ec7640 1 -- 172.21.15.27:0/2310973194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b412bfa0 msgr2=0x7fa2b4124430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7f558e05b640 1 --2- 172.21.15.27:0/4120284672 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f558812e340 0x7f5588070d90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5580032000 tx=0x7f5580009540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:18.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7fa2b8ec7640 1 --2- 172.21.15.27:0/2310973194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b412bfa0 0x7fa2b4124430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7fa2b8ec7640 1 -- 172.21.15.27:0/2310973194 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b41152b0 con 0x7fa2b412bb80 2024-09-21T03:27:18.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7f557e7fc640 1 -- 172.21.15.27:0/4120284672 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5580034070 con 0x7f558812e340 2024-09-21T03:27:18.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7f557e7fc640 1 -- 172.21.15.27:0/4120284672 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5580043a70 con 0x7f558812e340 2024-09-21T03:27:18.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7f558fabe640 1 -- 172.21.15.27:0/4120284672 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5588072a40 con 0x7f558812e340 2024-09-21T03:27:18.049 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7fa2b8ec7640 1 --2- 172.21.15.27:0/2310973194 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa2b412bb80 0x7fa2b4123ef0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400c520 tx=0x7fa2a400c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7f557e7fc640 1 -- 172.21.15.27:0/4120284672 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5580004250 con 0x7f558812e340 2024-09-21T03:27:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7f558fabe640 1 -- 172.21.15.27:0/4120284672 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5588135b30 con 0x7f558812e340 2024-09-21T03:27:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7fa2a9ffb640 1 -- 172.21.15.27:0/2310973194 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4017070 con 0x7fa2b412bb80 2024-09-21T03:27:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7fa2bb92c640 1 -- 172.21.15.27:0/2310973194 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b4124970 con 0x7fa2b412bb80 2024-09-21T03:27:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7fa2a9ffb640 1 -- 172.21.15.27:0/2310973194 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fa2a40035f0 con 0x7fa2b412bb80 2024-09-21T03:27:18.050 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.041+0000 7fa2a9ffb640 1 -- 172.21.15.27:0/2310973194 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4003a20 con 0x7fa2b412bb80 2024-09-21T03:27:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.042+0000 7f558fabe640 1 -- 172.21.15.27:0/4120284672 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5588071e40 con 0x7f558812e340 2024-09-21T03:27:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.043+0000 7fa2bb92c640 1 -- 172.21.15.27:0/2310973194 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b41c9b30 con 0x7fa2b412bb80 2024-09-21T03:27:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.043+0000 7f557e7fc640 1 -- 172.21.15.27:0/4120284672 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f5580037020 con 0x7f558812e340 2024-09-21T03:27:18.051 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.043+0000 7f557e7fc640 1 -- 172.21.15.27:0/4120284672 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f5580087170 con 0x7f558812e340 2024-09-21T03:27:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.043+0000 7fa2a9ffb640 1 -- 172.21.15.27:0/2310973194 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7fa2a401a070 con 0x7fa2b412bb80 2024-09-21T03:27:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.044+0000 7fa2977fe640 1 -- 172.21.15.27:0/2310973194 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa290005290 con 0x7fa2b412bb80 2024-09-21T03:27:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.047+0000 7fa2a9ffb640 1 -- 172.21.15.27:0/2310973194 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7fa2a403bdb0 con 0x7fa2b412bb80 2024-09-21T03:27:18.052 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.047+0000 7f557e7fc640 1 -- 172.21.15.27:0/4120284672 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f5580067df0 con 0x7f558812e340 2024-09-21T03:27:18.053 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.051+0000 7fa2a9ffb640 1 -- 172.21.15.27:0/2310973194 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7fa2a403c010 con 0x7fa2b412bb80 2024-09-21T03:27:18.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.078+0000 7f613599d640 1 -- 172.21.15.27:0/2432793204 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6130127d60 msgr2=0x7f6130132ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.078+0000 7f613599d640 1 --2- 172.21.15.27:0/2432793204 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6130127d60 0x7f6130132ac0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6128005fd0 tx=0x7f612802a780 comp rx=0 tx=0).stop 2024-09-21T03:27:18.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.078+0000 7f613599d640 1 -- 172.21.15.27:0/2432793204 shutdown_connections 2024-09-21T03:27:18.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.078+0000 7f613599d640 1 --2- 172.21.15.27:0/2432793204 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6130127d60 0x7f6130132ac0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.078+0000 7f613599d640 1 --2- 172.21.15.27:0/2432793204 >> v2:172.21.15.27:3300/0 conn(0x7f613012ae30 0x7f613012b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.078+0000 7f613599d640 1 -- 172.21.15.27:0/2432793204 >> 172.21.15.27:0/2432793204 conn(0x7f613009ce00 msgr2=0x7f613009d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:18.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.078+0000 7f613599d640 1 -- 172.21.15.27:0/2432793204 shutdown_connections 2024-09-21T03:27:18.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.078+0000 7f613599d640 1 -- 172.21.15.27:0/2432793204 wait complete. 2024-09-21T03:27:18.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.079+0000 7f613599d640 1 Processor -- start 2024-09-21T03:27:18.081 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.079+0000 7f613599d640 1 -- start start 2024-09-21T03:27:18.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613599d640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6130127200 0x7f6130072720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613599d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6130127d60 0x7f6130072c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613599d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f613012ae30 0x7f61300731a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613599d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f61301336c0 con 0x7f6130127d60 2024-09-21T03:27:18.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613599d640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f6130133540 con 0x7f613012ae30 2024-09-21T03:27:18.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613599d640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f6130133840 con 0x7f6130127200 2024-09-21T03:27:18.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f612ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6130127d60 0x7f6130072c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:18.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f612ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6130127d60 0x7f6130072c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40888/0 (socket says 172.21.15.27:40888) 2024-09-21T03:27:18.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f612ffff640 1 -- 172.21.15.27:0/351828030 learned_addr learned my addr 172.21.15.27:0/351828030 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:18.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613499b640 1 --2- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6130127200 0x7f6130072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613519c640 1 --2- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f613012ae30 0x7f61300731a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613499b640 1 -- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f613012ae30 msgr2=0x7f61300731a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613499b640 1 --2- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f613012ae30 0x7f61300731a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613499b640 1 -- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6130127d60 msgr2=0x7f6130072c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613499b640 1 --2- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6130127d60 0x7f6130072c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f613499b640 1 -- 172.21.15.27:0/351828030 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f613012bc30 con 0x7f6130127200 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.080+0000 7f612ffff640 1 --2- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6130127d60 0x7f6130072c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.081+0000 7f613499b640 1 --2- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6130127200 0x7f6130072720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6124009e60 tx=0x7f612400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.081+0000 7f612d7fa640 1 -- 172.21.15.27:0/351828030 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6124024070 con 0x7f6130127200 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.081+0000 7f612d7fa640 1 -- 172.21.15.27:0/351828030 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f612400c9a0 con 0x7f6130127200 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.081+0000 7f612d7fa640 1 -- 172.21.15.27:0/351828030 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612400cdb0 con 0x7f6130127200 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.081+0000 7f613599d640 1 -- 172.21.15.27:0/351828030 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6130124eb0 con 0x7f6130127200 2024-09-21T03:27:18.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.081+0000 7f613599d640 1 -- 172.21.15.27:0/351828030 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f613012ce80 con 0x7f6130127200 2024-09-21T03:27:18.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.082+0000 7f613599d640 1 -- 172.21.15.27:0/351828030 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f613012b210 con 0x7f6130127200 2024-09-21T03:27:18.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.082+0000 7f612d7fa640 1 -- 172.21.15.27:0/351828030 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f6124027070 con 0x7f6130127200 2024-09-21T03:27:18.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.083+0000 7f612d7fa640 1 -- 172.21.15.27:0/351828030 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f6124066640 con 0x7f6130127200 2024-09-21T03:27:18.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.087+0000 7f612d7fa640 1 -- 172.21.15.27:0/351828030 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f6124045830 con 0x7f6130127200 2024-09-21T03:27:18.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.089+0000 7f38ea7fc640 1 -- 172.21.15.27:0/1796663436 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e4032d10 con 0x7f38ec131210 2024-09-21T03:27:18.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.090+0000 7f38f220d640 1 -- 172.21.15.27:0/1796663436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec131210 msgr2=0x7f38ec133600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.090+0000 7f38f220d640 1 --2- 172.21.15.27:0/1796663436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec131210 0x7f38ec133600 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f38e4005fd0 tx=0x7f38e402a780 comp rx=0 tx=0).stop 2024-09-21T03:27:18.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.090+0000 7f38f220d640 1 -- 172.21.15.27:0/1796663436 shutdown_connections 2024-09-21T03:27:18.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.090+0000 7f38f220d640 1 --2- 172.21.15.27:0/1796663436 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec131210 0x7f38ec133600 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.090+0000 7f38f220d640 1 --2- 172.21.15.27:0/1796663436 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f38ec128f20 0x7f38ec130cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.090+0000 7f38f220d640 1 -- 172.21.15.27:0/1796663436 >> 172.21.15.27:0/1796663436 conn(0x7f38ec09c590 msgr2=0x7f38ec09c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:18.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.090+0000 7f38f220d640 1 -- 172.21.15.27:0/1796663436 shutdown_connections 2024-09-21T03:27:18.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.090+0000 7f38f220d640 1 -- 172.21.15.27:0/1796663436 wait complete. 2024-09-21T03:27:18.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.091+0000 7f38f220d640 1 Processor -- start 2024-09-21T03:27:18.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.091+0000 7f38f220d640 1 -- start start 2024-09-21T03:27:18.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.092+0000 7f38f220d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f38ec128430 0x7f38ec121df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.092+0000 7f38f220d640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec128f20 0x7f38ec122330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.092+0000 7f38f220d640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f38ec131210 0x7f38ec12ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.092+0000 7f38f220d640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec1348f0 con 0x7f38ec128f20 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.092+0000 7f38f220d640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f38ec134770 con 0x7f38ec128430 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.092+0000 7f38f220d640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f38ec134a70 con 0x7f38ec131210 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.092+0000 7f38eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec128f20 0x7f38ec122330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.092+0000 7f38eaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec128f20 0x7f38ec122330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40906/0 (socket says 172.21.15.27:40906) 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.092+0000 7f38eaffd640 1 -- 172.21.15.27:0/2773778419 learned_addr learned my addr 172.21.15.27:0/2773778419 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.092+0000 7f38eb7fe640 1 --2- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f38ec128430 0x7f38ec121df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38ebfff640 1 --2- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f38ec131210 0x7f38ec12ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38eb7fe640 1 -- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f38ec131210 msgr2=0x7f38ec12ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38eb7fe640 1 --2- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f38ec131210 0x7f38ec12ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38eb7fe640 1 -- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec128f20 msgr2=0x7f38ec122330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38eb7fe640 1 --2- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec128f20 0x7f38ec122330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38eb7fe640 1 -- 172.21.15.27:0/2773778419 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ec12eb40 con 0x7f38ec128430 2024-09-21T03:27:18.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38eaffd640 1 --2- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec128f20 0x7f38ec122330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:18.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38eb7fe640 1 --2- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f38ec128430 0x7f38ec121df0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00c3f0 tx=0x7f38dc00c7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:18.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38d3fff640 1 -- 172.21.15.27:0/2773778419 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38dc011790 con 0x7f38ec128430 2024-09-21T03:27:18.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38d3fff640 1 -- 172.21.15.27:0/2773778419 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f38dc017070 con 0x7f38ec128430 2024-09-21T03:27:18.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38f220d640 1 -- 172.21.15.27:0/2773778419 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38ec12e180 con 0x7f38ec128430 2024-09-21T03:27:18.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38d3fff640 1 -- 172.21.15.27:0/2773778419 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38dc011cc0 con 0x7f38ec128430 2024-09-21T03:27:18.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.093+0000 7f38f220d640 1 -- 172.21.15.27:0/2773778419 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ec12f260 con 0x7f38ec128430 2024-09-21T03:27:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.095+0000 7f38f220d640 1 -- 172.21.15.27:0/2773778419 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38ec09a1f0 con 0x7f38ec128430 2024-09-21T03:27:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.095+0000 7f38d3fff640 1 -- 172.21.15.27:0/2773778419 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f38dc01c460 con 0x7f38ec128430 2024-09-21T03:27:18.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.096+0000 7f38d3fff640 1 -- 172.21.15.27:0/2773778419 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 2868+0+0 (secure 0 0 0) 0x7f38dc05d700 con 0x7f38ec128430 2024-09-21T03:27:18.101 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.099+0000 7f38d3fff640 1 -- 172.21.15.27:0/2773778419 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f38ec09a1f0 con 0x7f38ec128430 2024-09-21T03:27:18.221 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.219+0000 7fa2977fe640 1 -- 172.21.15.27:0/2310973194 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa290005850 con 0x7fa2b412bb80 2024-09-21T03:27:18.222 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.220+0000 7fa2a9ffb640 1 -- 172.21.15.27:0/2310973194 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa2a405bc70 con 0x7fa2b412bb80 2024-09-21T03:27:18.222 INFO:teuthology.orchestra.run.smithi027.stdout:47244640259 2024-09-21T03:27:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.222+0000 7fa2977fe640 1 -- 172.21.15.27:0/2310973194 >> v1:172.21.15.27:6800/1098903051 conn(0x7fa298047fd0 legacy=0x7fa29804a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.222+0000 7fa2977fe640 1 -- 172.21.15.27:0/2310973194 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa2b412bb80 msgr2=0x7fa2b4123ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.222+0000 7fa2977fe640 1 --2- 172.21.15.27:0/2310973194 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa2b412bb80 0x7fa2b4123ef0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa2a400c520 tx=0x7fa2a400c9e0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.222+0000 7fa2977fe640 1 -- 172.21.15.27:0/2310973194 shutdown_connections 2024-09-21T03:27:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.222+0000 7fa2977fe640 1 --2- 172.21.15.27:0/2310973194 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2b412bfa0 0x7fa2b4124430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.222+0000 7fa2977fe640 1 --2- 172.21.15.27:0/2310973194 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fa2b412bb80 0x7fa2b4123ef0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.225 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.222+0000 7fa2977fe640 1 --2- 172.21.15.27:0/2310973194 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fa2b4072a60 0x7fa2b41239b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.222+0000 7fa2977fe640 1 -- 172.21.15.27:0/2310973194 >> 172.21.15.27:0/2310973194 conn(0x7fa2b409c350 msgr2=0x7fa2b406fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.223+0000 7fa2977fe640 1 -- 172.21.15.27:0/2310973194 shutdown_connections 2024-09-21T03:27:18.226 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.223+0000 7fa2977fe640 1 -- 172.21.15.27:0/2310973194 wait complete. 2024-09-21T03:27:18.240 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.0 2024-09-21T03:27:18.244 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.243+0000 7f613599d640 1 -- 172.21.15.27:0/351828030 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f613004ffb0 con 0x7f6130127200 2024-09-21T03:27:18.245 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.243+0000 7f612d7fa640 1 -- 172.21.15.27:0/351828030 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6124045a90 con 0x7f6130127200 2024-09-21T03:27:18.245 INFO:teuthology.orchestra.run.smithi027.stdout:47244640258 2024-09-21T03:27:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.245+0000 7f613599d640 1 -- 172.21.15.27:0/351828030 >> v1:172.21.15.27:6800/1098903051 conn(0x7f6114047fd0 legacy=0x7f611404a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.245+0000 7f613599d640 1 -- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6130127200 msgr2=0x7f6130072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.245+0000 7f613599d640 1 --2- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6130127200 0x7f6130072720 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6124009e60 tx=0x7f612400edf0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.246+0000 7f613599d640 1 -- 172.21.15.27:0/351828030 shutdown_connections 2024-09-21T03:27:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.246+0000 7f613599d640 1 --2- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f613012ae30 0x7f61300731a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.246+0000 7f613599d640 1 --2- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6130127d60 0x7f6130072c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.246+0000 7f613599d640 1 --2- 172.21.15.27:0/351828030 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6130127200 0x7f6130072720 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.246+0000 7f613599d640 1 -- 172.21.15.27:0/351828030 >> 172.21.15.27:0/351828030 conn(0x7f613009ce00 msgr2=0x7f61301268e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:18.248 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.246+0000 7f613599d640 1 -- 172.21.15.27:0/351828030 shutdown_connections 2024-09-21T03:27:18.249 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.246+0000 7f613599d640 1 -- 172.21.15.27:0/351828030 wait complete. 2024-09-21T03:27:18.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-09-21T03:27:18.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.262+0000 7f558fabe640 1 -- 172.21.15.27:0/4120284672 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f5588126a40 con 0x7f558812e340 2024-09-21T03:27:18.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.262+0000 7f557e7fc640 1 -- 172.21.15.27:0/4120284672 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5580000fb0 con 0x7f558812e340 2024-09-21T03:27:18.264 INFO:teuthology.orchestra.run.smithi027.stdout:47244640258 2024-09-21T03:27:18.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.264+0000 7f558fabe640 1 -- 172.21.15.27:0/4120284672 >> v1:172.21.15.27:6800/1098903051 conn(0x7f556c047fd0 legacy=0x7f556c04a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.264+0000 7f558fabe640 1 -- 172.21.15.27:0/4120284672 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f558812e340 msgr2=0x7f5588070d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.264+0000 7f558fabe640 1 --2- 172.21.15.27:0/4120284672 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f558812e340 0x7f5588070d90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5580032000 tx=0x7f5580009540 comp rx=0 tx=0).stop 2024-09-21T03:27:18.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.264+0000 7f558fabe640 1 -- 172.21.15.27:0/4120284672 shutdown_connections 2024-09-21T03:27:18.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.264+0000 7f558fabe640 1 --2- 172.21.15.27:0/4120284672 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f558812e340 0x7f5588070d90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.264+0000 7f558fabe640 1 --2- 172.21.15.27:0/4120284672 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5588098830 0x7f5588070850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.264+0000 7f558fabe640 1 --2- 172.21.15.27:0/4120284672 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5588097f10 0x7f5588070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.264+0000 7f558fabe640 1 -- 172.21.15.27:0/4120284672 >> 172.21.15.27:0/4120284672 conn(0x7f558809e590 msgr2=0x7f558809b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:18.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.264+0000 7f558fabe640 1 -- 172.21.15.27:0/4120284672 shutdown_connections 2024-09-21T03:27:18.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.264+0000 7f558fabe640 1 -- 172.21.15.27:0/4120284672 wait complete. 2024-09-21T03:27:18.277 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.2 2024-09-21T03:27:18.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.306+0000 7f38f220d640 1 -- 172.21.15.27:0/2773778419 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f38ec067d30 con 0x7f38ec128430 2024-09-21T03:27:18.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.306+0000 7f38d3fff640 1 -- 172.21.15.27:0/2773778419 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f38dc000fe0 con 0x7f38ec128430 2024-09-21T03:27:18.308 INFO:teuthology.orchestra.run.smithi027.stdout:38654705668 2024-09-21T03:27:18.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.308+0000 7f38f220d640 1 -- 172.21.15.27:0/2773778419 >> v1:172.21.15.27:6800/1098903051 conn(0x7f38d8047fd0 legacy=0x7f38d804a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.308+0000 7f38f220d640 1 -- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f38ec128430 msgr2=0x7f38ec121df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:18.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.308+0000 7f38f220d640 1 --2- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f38ec128430 0x7f38ec121df0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00c3f0 tx=0x7f38dc00c7f0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.308+0000 7f38f220d640 1 -- 172.21.15.27:0/2773778419 shutdown_connections 2024-09-21T03:27:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.308+0000 7f38f220d640 1 --2- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f38ec131210 0x7f38ec12ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.308+0000 7f38f220d640 1 --2- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38ec128f20 0x7f38ec122330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.308+0000 7f38f220d640 1 --2- 172.21.15.27:0/2773778419 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f38ec128430 0x7f38ec121df0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.308+0000 7f38f220d640 1 -- 172.21.15.27:0/2773778419 >> 172.21.15.27:0/2773778419 conn(0x7f38ec09c590 msgr2=0x7f38ec127350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.308+0000 7f38f220d640 1 -- 172.21.15.27:0/2773778419 shutdown_connections 2024-09-21T03:27:18.311 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:18.308+0000 7f38f220d640 1 -- 172.21.15.27:0/2773778419 wait complete. 2024-09-21T03:27:18.320 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.1 2024-09-21T03:27:18.321 DEBUG:teuthology.parallel:result is None 2024-09-21T03:27:19.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T03:27:19.263 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T03:27:19.278 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T03:27:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.363+0000 7facb7067640 1 -- 172.21.15.27:0/668768824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb0098830 msgr2=0x7facb012dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.363+0000 7facb7067640 1 --2- 172.21.15.27:0/668768824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb0098830 0x7facb012dc30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fac9c005fd0 tx=0x7fac9c02a410 comp rx=0 tx=0).stop 2024-09-21T03:27:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.363+0000 7facb7067640 1 -- 172.21.15.27:0/668768824 shutdown_connections 2024-09-21T03:27:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.363+0000 7facb7067640 1 --2- 172.21.15.27:0/668768824 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facb012e170 0x7facb012e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.363+0000 7facb7067640 1 --2- 172.21.15.27:0/668768824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb0098830 0x7facb012dc30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.363+0000 7facb7067640 1 --2- 172.21.15.27:0/668768824 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facb0097f10 0x7facb00982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.363+0000 7facb7067640 1 -- 172.21.15.27:0/668768824 >> 172.21.15.27:0/668768824 conn(0x7facb009e590 msgr2=0x7facb009e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.363+0000 7facb7067640 1 -- 172.21.15.27:0/668768824 shutdown_connections 2024-09-21T03:27:19.365 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.363+0000 7facb7067640 1 -- 172.21.15.27:0/668768824 wait complete. 2024-09-21T03:27:19.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.364+0000 7facb7067640 1 Processor -- start 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.364+0000 7facb7067640 1 -- start start 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.365+0000 7facb7067640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb0097f10 0x7facb01c3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.365+0000 7facb7067640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facb0098830 0x7facb01c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.365+0000 7facb7067640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facb012e170 0x7facb01c4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.365+0000 7facb7067640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7facb0135660 con 0x7facb0097f10 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.365+0000 7facb7067640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7facb01354e0 con 0x7facb0098830 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.365+0000 7facb7067640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7facb01357e0 con 0x7facb012e170 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.365+0000 7facb4e03640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb0097f10 0x7facb01c3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.365+0000 7facb5604640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facb012e170 0x7facb01c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.365+0000 7facb4e03640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb0097f10 0x7facb01c3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40932/0 (socket says 172.21.15.27:40932) 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.365+0000 7facb4e03640 1 -- 172.21.15.27:0/2622727852 learned_addr learned my addr 172.21.15.27:0/2622727852 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.365+0000 7facaffff640 1 --2- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facb0098830 0x7facb01c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facb5604640 1 -- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facb0098830 msgr2=0x7facb01c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facb5604640 1 --2- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facb0098830 0x7facb01c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facb5604640 1 -- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb0097f10 msgr2=0x7facb01c3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facb5604640 1 --2- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb0097f10 0x7facb01c3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facb5604640 1 -- 172.21.15.27:0/2622727852 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb012f140 con 0x7facb012e170 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facb4e03640 1 --2- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb0097f10 0x7facb01c3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facaffff640 1 --2- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facb0098830 0x7facb01c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facb5604640 1 --2- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facb012e170 0x7facb01c4540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faca800ec50 tx=0x7faca800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facad7fa640 1 -- 172.21.15.27:0/2622727852 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca8003b70 con 0x7facb012e170 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facad7fa640 1 -- 172.21.15.27:0/2622727852 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7faca8003d10 con 0x7facb012e170 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facb7067640 1 -- 172.21.15.27:0/2622727852 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb0073720 con 0x7facb012e170 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facad7fa640 1 -- 172.21.15.27:0/2622727852 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca8004280 con 0x7facb012e170 2024-09-21T03:27:19.369 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.366+0000 7facb7067640 1 -- 172.21.15.27:0/2622727852 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7facb00708d0 con 0x7facb012e170 2024-09-21T03:27:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.368+0000 7facb7067640 1 -- 172.21.15.27:0/2622727852 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb012dc30 con 0x7facb012e170 2024-09-21T03:27:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.368+0000 7facad7fa640 1 -- 172.21.15.27:0/2622727852 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7faca8004420 con 0x7facb012e170 2024-09-21T03:27:19.370 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.368+0000 7facad7fa640 1 -- 172.21.15.27:0/2622727852 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(19..19 src has 1..19) ==== 2840+0+0 (secure 0 0 0) 0x7faca805e450 con 0x7facb012e170 2024-09-21T03:27:19.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.372+0000 7facad7fa640 1 -- 172.21.15.27:0/2622727852 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7faca800ce10 con 0x7facb012e170 2024-09-21T03:27:19.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.402+0000 7f2a6cb25640 1 -- 172.21.15.27:0/1705307428 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a68098830 msgr2=0x7f2a6812dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.402+0000 7f2a6cb25640 1 --2- 172.21.15.27:0/1705307428 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a68098830 0x7f2a6812dc30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009830 tx=0x7f2a5002a360 comp rx=0 tx=0).stop 2024-09-21T03:27:19.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.402+0000 7f2a6cb25640 1 -- 172.21.15.27:0/1705307428 shutdown_connections 2024-09-21T03:27:19.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.402+0000 7f2a6cb25640 1 --2- 172.21.15.27:0/1705307428 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a6812e170 0x7f2a6812e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.402+0000 7f2a6cb25640 1 --2- 172.21.15.27:0/1705307428 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a68098830 0x7f2a6812dc30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.402+0000 7f2a6cb25640 1 --2- 172.21.15.27:0/1705307428 >> v2:172.21.15.27:3300/0 conn(0x7f2a68097f10 0x7f2a680982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.402+0000 7f2a6cb25640 1 -- 172.21.15.27:0/1705307428 >> 172.21.15.27:0/1705307428 conn(0x7f2a6809e590 msgr2=0x7f2a6809e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:19.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.402+0000 7f2a6cb25640 1 -- 172.21.15.27:0/1705307428 shutdown_connections 2024-09-21T03:27:19.404 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.402+0000 7f2a6cb25640 1 -- 172.21.15.27:0/1705307428 wait complete. 2024-09-21T03:27:19.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.403+0000 7f5338648640 1 -- 172.21.15.27:0/537689169 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5330072a60 msgr2=0x7f5330128aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.403+0000 7f5338648640 1 --2- 172.21.15.27:0/537689169 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5330072a60 0x7f5330128aa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5328005fd0 tx=0x7f532802a780 comp rx=0 tx=0).stop 2024-09-21T03:27:19.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.403+0000 7f5338648640 1 -- 172.21.15.27:0/537689169 shutdown_connections 2024-09-21T03:27:19.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.403+0000 7f5338648640 1 --2- 172.21.15.27:0/537689169 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5330072a60 0x7f5330128aa0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.403+0000 7f5338648640 1 --2- 172.21.15.27:0/537689169 >> v2:172.21.15.27:3300/0 conn(0x7f533012bb80 0x7f533012c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.403+0000 7f5338648640 1 -- 172.21.15.27:0/537689169 >> 172.21.15.27:0/537689169 conn(0x7f533009c350 msgr2=0x7f533009c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:19.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.403+0000 7f5338648640 1 -- 172.21.15.27:0/537689169 shutdown_connections 2024-09-21T03:27:19.405 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.403+0000 7f5338648640 1 -- 172.21.15.27:0/537689169 wait complete. 2024-09-21T03:27:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.403+0000 7f2a6cb25640 1 Processor -- start 2024-09-21T03:27:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.403+0000 7f2a6cb25640 1 -- start start 2024-09-21T03:27:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5338648640 1 Processor -- start 2024-09-21T03:27:19.406 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5338648640 1 -- start start 2024-09-21T03:27:19.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f2a6cb25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a68097f10 0x7f2a681c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f2a6cb25640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a68098830 0x7f2a681c4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f2a6cb25640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a6812e170 0x7f2a681c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.407 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5338648640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5330072a60 0x7f53301c1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f2a6cb25640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2a681356c0 con 0x7f2a68097f10 2024-09-21T03:27:19.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f2a6cb25640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f2a68135540 con 0x7f2a6812e170 2024-09-21T03:27:19.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f2a6cb25640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f2a68135840 con 0x7f2a68098830 2024-09-21T03:27:19.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5338648640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f533012bb80 0x7f53301c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5338648640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f533012bfa0 0x7f533012f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.408 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f2a5ffff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a68098830 0x7f2a681c4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5338648640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f53301346d0 con 0x7f533012bfa0 2024-09-21T03:27:19.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5338648640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f5330134550 con 0x7f5330072a60 2024-09-21T03:27:19.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5338648640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f5330134850 con 0x7f533012bb80 2024-09-21T03:27:19.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5335be3640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f533012bb80 0x7f53301c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5335be3640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f533012bb80 0x7f53301c22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:47308/0 (socket says 172.21.15.27:47308) 2024-09-21T03:27:19.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5335be3640 1 -- 172.21.15.27:0/772385106 learned_addr learned my addr 172.21.15.27:0/772385106 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:19.409 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f53363e4640 1 --2- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5330072a60 0x7f53301c1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f2a6659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a68097f10 0x7f2a681c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f2a5ffff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a68098830 0x7f2a681c4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:47306/0 (socket says 172.21.15.27:47306) 2024-09-21T03:27:19.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f2a5ffff640 1 -- 172.21.15.27:0/41811957 learned_addr learned my addr 172.21.15.27:0/41811957 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:19.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f2a66d9d640 1 --2- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a6812e170 0x7f2a681c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.410 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.404+0000 7f5336be5640 1 --2- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f533012bfa0 0x7f533012f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f53363e4640 1 -- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f533012bb80 msgr2=0x7f53301c22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f2a5ffff640 1 -- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a6812e170 msgr2=0x7f2a681c45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f2a5ffff640 1 --2- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a6812e170 0x7f2a681c45a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f2a5ffff640 1 -- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a68097f10 msgr2=0x7f2a681c3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f2a5ffff640 1 --2- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a68097f10 0x7f2a681c3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.411 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f2a5ffff640 1 -- 172.21.15.27:0/41811957 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6806dda0 con 0x7f2a68098830 2024-09-21T03:27:19.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f53363e4640 1 --2- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f533012bb80 0x7f53301c22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f53363e4640 1 -- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f533012bfa0 msgr2=0x7f533012f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.412 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f2a6659c640 1 --2- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a68097f10 0x7f2a681c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:19.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f53363e4640 1 --2- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f533012bfa0 0x7f533012f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f53363e4640 1 -- 172.21.15.27:0/772385106 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5330130f60 con 0x7f5330072a60 2024-09-21T03:27:19.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f5335be3640 1 --2- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f533012bb80 0x7f53301c22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:19.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f53363e4640 1 --2- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5330072a60 0x7f53301c1d90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f532c0016f0 tx=0x7f532c001bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:19.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f5326ffd640 1 -- 172.21.15.27:0/772385106 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c010830 con 0x7f5330072a60 2024-09-21T03:27:19.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f5338648640 1 -- 172.21.15.27:0/772385106 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5330132bf0 con 0x7f5330072a60 2024-09-21T03:27:19.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f5326ffd640 1 -- 172.21.15.27:0/772385106 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f532c0109d0 con 0x7f5330072a60 2024-09-21T03:27:19.413 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f2a5ffff640 1 --2- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a68098830 0x7f2a681c4060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009780 tx=0x7f2a50033b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:19.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f5338648640 1 -- 172.21.15.27:0/772385106 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f53301317f0 con 0x7f5330072a60 2024-09-21T03:27:19.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.405+0000 7f5326ffd640 1 -- 172.21.15.27:0/772385106 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c010e20 con 0x7f5330072a60 2024-09-21T03:27:19.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.406+0000 7f2a5effd640 1 -- 172.21.15.27:0/41811957 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5003f020 con 0x7f2a68098830 2024-09-21T03:27:19.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.406+0000 7f2a5effd640 1 -- 172.21.15.27:0/41811957 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a500035f0 con 0x7f2a68098830 2024-09-21T03:27:19.414 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.406+0000 7f2a5effd640 1 -- 172.21.15.27:0/41811957 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50037040 con 0x7f2a68098830 2024-09-21T03:27:19.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.406+0000 7f2a6cb25640 1 -- 172.21.15.27:0/41811957 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a68133400 con 0x7f2a68098830 2024-09-21T03:27:19.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.406+0000 7f5338648640 1 -- 172.21.15.27:0/772385106 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f533012c380 con 0x7f5330072a60 2024-09-21T03:27:19.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.406+0000 7f2a6cb25640 1 -- 172.21.15.27:0/41811957 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a68131930 con 0x7f2a68098830 2024-09-21T03:27:19.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.407+0000 7f5326ffd640 1 -- 172.21.15.27:0/772385106 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f532c0173b0 con 0x7f5330072a60 2024-09-21T03:27:19.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.407+0000 7f5326ffd640 1 -- 172.21.15.27:0/772385106 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 2840+0+0 (secure 0 0 0) 0x7f532c010bc0 con 0x7f5330072a60 2024-09-21T03:27:19.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.407+0000 7f2a5effd640 1 -- 172.21.15.27:0/41811957 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f2a50042020 con 0x7f2a68098830 2024-09-21T03:27:19.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.408+0000 7f2a5effd640 1 -- 172.21.15.27:0/41811957 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(19..19 src has 1..19) ==== 2840+0+0 (secure 0 0 0) 0x7f2a5007d300 con 0x7f2a68098830 2024-09-21T03:27:19.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.408+0000 7f2a6cb25640 1 -- 172.21.15.27:0/41811957 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a68133cc0 con 0x7f2a68098830 2024-09-21T03:27:19.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.411+0000 7f5326ffd640 1 -- 172.21.15.27:0/772385106 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f532c00c070 con 0x7f5330072a60 2024-09-21T03:27:19.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.412+0000 7f2a5effd640 1 -- 172.21.15.27:0/41811957 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f2a50049020 con 0x7f2a68098830 2024-09-21T03:27:19.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.527+0000 7facb7067640 1 -- 172.21.15.27:0/2622727852 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7facb0067d30 con 0x7facb012e170 2024-09-21T03:27:19.529 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.528+0000 7facad7fa640 1 -- 172.21.15.27:0/2622727852 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faca805d5c0 con 0x7facb012e170 2024-09-21T03:27:19.529 INFO:teuthology.orchestra.run.smithi027.stdout:47244640260 2024-09-21T03:27:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.529+0000 7facb7067640 1 -- 172.21.15.27:0/2622727852 >> v1:172.21.15.27:6800/1098903051 conn(0x7fac94048000 legacy=0x7fac9404a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.529+0000 7facb7067640 1 -- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facb012e170 msgr2=0x7facb01c4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.529+0000 7facb7067640 1 --2- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facb012e170 0x7facb01c4540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faca800ec50 tx=0x7faca800c6a0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.529+0000 7facb7067640 1 -- 172.21.15.27:0/2622727852 shutdown_connections 2024-09-21T03:27:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.529+0000 7facb7067640 1 --2- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facb012e170 0x7facb01c4540 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.529+0000 7facb7067640 1 --2- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facb0098830 0x7facb01c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.529+0000 7facb7067640 1 --2- 172.21.15.27:0/2622727852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facb0097f10 0x7facb01c3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.529+0000 7facb7067640 1 -- 172.21.15.27:0/2622727852 >> 172.21.15.27:0/2622727852 conn(0x7facb009e590 msgr2=0x7facb012a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.529+0000 7facb7067640 1 -- 172.21.15.27:0/2622727852 shutdown_connections 2024-09-21T03:27:19.531 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.530+0000 7facb7067640 1 -- 172.21.15.27:0/2622727852 wait complete. 2024-09-21T03:27:19.542 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-09-21T03:27:19.542 DEBUG:teuthology.parallel:result is None 2024-09-21T03:27:19.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.562+0000 7f5338648640 1 -- 172.21.15.27:0/772385106 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f5330067d30 con 0x7f5330072a60 2024-09-21T03:27:19.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.562+0000 7f5326ffd640 1 -- 172.21.15.27:0/772385106 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f532c00e280 con 0x7f5330072a60 2024-09-21T03:27:19.564 INFO:teuthology.orchestra.run.smithi027.stdout:47244640259 2024-09-21T03:27:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.564+0000 7f5338648640 1 -- 172.21.15.27:0/772385106 >> v1:172.21.15.27:6800/1098903051 conn(0x7f53140481d0 legacy=0x7f531404a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.564+0000 7f5338648640 1 -- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5330072a60 msgr2=0x7f53301c1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.564+0000 7f5338648640 1 --2- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5330072a60 0x7f53301c1d90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f532c0016f0 tx=0x7f532c001bb0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.564+0000 7f5338648640 1 -- 172.21.15.27:0/772385106 shutdown_connections 2024-09-21T03:27:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.564+0000 7f5338648640 1 --2- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f533012bfa0 0x7f533012f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.564+0000 7f5338648640 1 --2- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f533012bb80 0x7f53301c22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.566 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.564+0000 7f5338648640 1 --2- 172.21.15.27:0/772385106 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5330072a60 0x7f53301c1d90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.564+0000 7f5338648640 1 -- 172.21.15.27:0/772385106 >> 172.21.15.27:0/772385106 conn(0x7f533009c350 msgr2=0x7f533012b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.565+0000 7f5338648640 1 -- 172.21.15.27:0/772385106 shutdown_connections 2024-09-21T03:27:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.565+0000 7f5338648640 1 -- 172.21.15.27:0/772385106 wait complete. 2024-09-21T03:27:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.565+0000 7f2a6cb25640 1 -- 172.21.15.27:0/41811957 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f2a68067d30 con 0x7f2a68098830 2024-09-21T03:27:19.567 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.566+0000 7f2a5effd640 1 -- 172.21.15.27:0/41811957 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2a5005c5d0 con 0x7f2a68098830 2024-09-21T03:27:19.567 INFO:teuthology.orchestra.run.smithi027.stdout:47244640259 2024-09-21T03:27:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.569+0000 7f2a6cb25640 1 -- 172.21.15.27:0/41811957 >> v1:172.21.15.27:6800/1098903051 conn(0x7f2a48048020 legacy=0x7f2a4804a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.569+0000 7f2a6cb25640 1 -- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a68098830 msgr2=0x7f2a681c4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.569+0000 7f2a6cb25640 1 --2- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a68098830 0x7f2a681c4060 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2a50009780 tx=0x7f2a50033b50 comp rx=0 tx=0).stop 2024-09-21T03:27:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.569+0000 7f2a6cb25640 1 -- 172.21.15.27:0/41811957 shutdown_connections 2024-09-21T03:27:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.569+0000 7f2a6cb25640 1 --2- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a6812e170 0x7f2a681c45a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.569+0000 7f2a6cb25640 1 --2- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a68098830 0x7f2a681c4060 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.569+0000 7f2a6cb25640 1 --2- 172.21.15.27:0/41811957 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a68097f10 0x7f2a681c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.570 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.569+0000 7f2a6cb25640 1 -- 172.21.15.27:0/41811957 >> 172.21.15.27:0/41811957 conn(0x7f2a6809e590 msgr2=0x7f2a6812b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.569+0000 7f2a6cb25640 1 -- 172.21.15.27:0/41811957 shutdown_connections 2024-09-21T03:27:19.571 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.569+0000 7f2a6cb25640 1 -- 172.21.15.27:0/41811957 wait complete. 2024-09-21T03:27:19.579 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.2 2024-09-21T03:27:19.579 DEBUG:teuthology.parallel:result is None 2024-09-21T03:27:19.582 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2024-09-21T03:27:19.582 DEBUG:teuthology.parallel:result is None 2024-09-21T03:27:19.582 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T03:27:19.582 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T03:27:19.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.724+0000 7f7fbb05a640 1 -- 172.21.15.27:0/3436615942 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7fb40929f0 msgr2=0x7f7fb408ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.724+0000 7f7fbb05a640 1 --2- 172.21.15.27:0/3436615942 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7fb40929f0 0x7f7fb408ffb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009890 tx=0x7f7fa402a4a0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.724+0000 7f7fbb05a640 1 -- 172.21.15.27:0/3436615942 shutdown_connections 2024-09-21T03:27:19.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.724+0000 7f7fbb05a640 1 --2- 172.21.15.27:0/3436615942 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7fb40929f0 0x7f7fb408ffb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.724+0000 7f7fbb05a640 1 --2- 172.21.15.27:0/3436615942 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7fb40920d0 0x7f7fb40924b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.724+0000 7f7fbb05a640 1 -- 172.21.15.27:0/3436615942 >> 172.21.15.27:0/3436615942 conn(0x7f7fb409d760 msgr2=0x7f7fb4099c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:19.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.724+0000 7f7fbb05a640 1 -- 172.21.15.27:0/3436615942 shutdown_connections 2024-09-21T03:27:19.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.724+0000 7f7fbb05a640 1 -- 172.21.15.27:0/3436615942 wait complete. 2024-09-21T03:27:19.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.725+0000 7f7fbb05a640 1 Processor -- start 2024-09-21T03:27:19.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.725+0000 7f7fbb05a640 1 -- start start 2024-09-21T03:27:19.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fbb05a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb40904f0 0x7f7fb406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fbb05a640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7fb40920d0 0x7f7fb406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fbb05a640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7fb40929f0 0x7f7fb406de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fbb05a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7fb4133750 con 0x7f7fb40904f0 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fbb05a640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f7fb4090dc0 con 0x7f7fb40929f0 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fbb05a640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f7fb41338d0 con 0x7f7fb40920d0 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fb8df6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb40904f0 0x7f7fb406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fabfff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7fb40920d0 0x7f7fb406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fb8df6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb40904f0 0x7f7fb406d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40970/0 (socket says 172.21.15.27:40970) 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fb8df6640 1 -- 172.21.15.27:0/151730557 learned_addr learned my addr 172.21.15.27:0/151730557 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fabfff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7fb40920d0 0x7f7fb406d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:47326/0 (socket says 172.21.15.27:47326) 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fb95f7640 1 --2- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7fb40929f0 0x7f7fb406de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fb95f7640 1 -- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7fb40920d0 msgr2=0x7f7fb406d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fb95f7640 1 --2- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7fb40920d0 0x7f7fb406d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fb95f7640 1 -- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb40904f0 msgr2=0x7f7fb406d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fb95f7640 1 --2- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb40904f0 0x7f7fb406d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fb95f7640 1 -- 172.21.15.27:0/151730557 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb412ab50 con 0x7f7fb40929f0 2024-09-21T03:27:19.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fb8df6640 1 --2- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb40904f0 0x7f7fb406d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:19.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fb95f7640 1 --2- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7fb40929f0 0x7f7fb406de70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00cb00 tx=0x7f7fac00a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:19.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fa97fa640 1 -- 172.21.15.27:0/151730557 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac016020 con 0x7f7fb40929f0 2024-09-21T03:27:19.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fa97fa640 1 -- 172.21.15.27:0/151730557 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7fac00ad90 con 0x7f7fb40929f0 2024-09-21T03:27:19.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fbb05a640 1 -- 172.21.15.27:0/151730557 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fb4071130 con 0x7f7fb40929f0 2024-09-21T03:27:19.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.726+0000 7f7fa97fa640 1 -- 172.21.15.27:0/151730557 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac012680 con 0x7f7fb40929f0 2024-09-21T03:27:19.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.727+0000 7f7fbb05a640 1 -- 172.21.15.27:0/151730557 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fb4071b10 con 0x7f7fb40929f0 2024-09-21T03:27:19.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.728+0000 7f7fbb05a640 1 -- 172.21.15.27:0/151730557 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fb4133370 con 0x7f7fb40929f0 2024-09-21T03:27:19.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.728+0000 7f7fa97fa640 1 -- 172.21.15.27:0/151730557 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f7fac019020 con 0x7f7fb40929f0 2024-09-21T03:27:19.731 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.728+0000 7f7fa97fa640 1 -- 172.21.15.27:0/151730557 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 2840+0+0 (secure 0 0 0) 0x7f7fac05dc60 con 0x7f7fb40929f0 2024-09-21T03:27:19.733 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.732+0000 7f7fa97fa640 1 -- 172.21.15.27:0/151730557 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f7fac020030 con 0x7f7fb40929f0 2024-09-21T03:27:19.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.877+0000 7f7fbb05a640 1 -- 172.21.15.27:0/151730557 --> v1:172.21.15.27:6800/1098903051 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7fb40924b0 con 0x7f7fa0047fd0 2024-09-21T03:27:19.880 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.879+0000 7f7fa97fa640 1 -- 172.21.15.27:0/151730557 <== mgr.4102 v1:172.21.15.27:6800/1098903051 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+32222 (unknown 2965378022 0 1129619846) 0x7f7fb40924b0 con 0x7f7fa0047fd0 2024-09-21T03:27:19.880 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:27:19.880 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-21T03:27:19.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.880+0000 7f7fbb05a640 1 -- 172.21.15.27:0/151730557 >> v1:172.21.15.27:6800/1098903051 conn(0x7f7fa0047fd0 legacy=0x7f7fa004a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.880+0000 7f7fbb05a640 1 -- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7fb40929f0 msgr2=0x7f7fb406de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.880+0000 7f7fbb05a640 1 --2- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7fb40929f0 0x7f7fb406de70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00cb00 tx=0x7f7fac00a060 comp rx=0 tx=0).stop 2024-09-21T03:27:19.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.880+0000 7f7fbb05a640 1 -- 172.21.15.27:0/151730557 shutdown_connections 2024-09-21T03:27:19.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.880+0000 7f7fbb05a640 1 --2- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7fb40929f0 0x7f7fb406de70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.880+0000 7f7fbb05a640 1 --2- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7fb40920d0 0x7f7fb406d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.880+0000 7f7fbb05a640 1 --2- 172.21.15.27:0/151730557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7fb40904f0 0x7f7fb406d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.880+0000 7f7fbb05a640 1 -- 172.21.15.27:0/151730557 >> 172.21.15.27:0/151730557 conn(0x7f7fb409d760 msgr2=0x7f7fb4097b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:19.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.881+0000 7f7fbb05a640 1 -- 172.21.15.27:0/151730557 shutdown_connections 2024-09-21T03:27:19.882 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.881+0000 7f7fbb05a640 1 -- 172.21.15.27:0/151730557 wait complete. 2024-09-21T03:27:19.892 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-21T03:27:19.501994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":114072,"kb_used_data":1776,"kb_used_omap":6,"kb_used_meta":112185,"kb_avail":377373288,"statfs":{"total":386547056640,"available":386430246912,"internally_reserved":0,"allocated":1818624,"data_stored":1353470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":114878231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":16,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000932"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.282474+0000","last_change":"2024-09-21T03:27:15.282474+0000","last_active":"2024-09-21T03:27:15.282474+0000","last_peered":"2024-09-21T03:27:15.282474+0000","last_clean":"2024-09-21T03:27:15.282474+0000","last_became_active":"2024-09-21T03:27:13.273870+0000","last_became_peered":"2024-09-21T03:27:13.273870+0000","last_unstale":"2024-09-21T03:27:15.282474+0000","last_undegraded":"2024-09-21T03:27:15.282474+0000","last_fullsized":"2024-09-21T03:27:15.282474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:22.300880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393285,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.287209+0000","last_change":"2024-09-21T03:27:15.287459+0000","last_active":"2024-09-21T03:27:15.287209+0000","last_peered":"2024-09-21T03:27:15.287209+0000","last_clean":"2024-09-21T03:27:15.287209+0000","last_became_active":"2024-09-21T03:27:13.271654+0000","last_became_peered":"2024-09-21T03:27:13.271654+0000","last_unstale":"2024-09-21T03:27:15.287209+0000","last_undegraded":"2024-09-21T03:27:15.287209+0000","last_fullsized":"2024-09-21T03:27:15.287209+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:45:52.098322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044937200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.282111+0000","last_change":"2024-09-21T03:27:15.282111+0000","last_active":"2024-09-21T03:27:15.282111+0000","last_peered":"2024-09-21T03:27:15.282111+0000","last_clean":"2024-09-21T03:27:15.282111+0000","last_became_active":"2024-09-21T03:27:13.273857+0000","last_became_peered":"2024-09-21T03:27:13.273857+0000","last_unstale":"2024-09-21T03:27:15.282111+0000","last_undegraded":"2024-09-21T03:27:15.282111+0000","last_fullsized":"2024-09-21T03:27:15.282111+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:47:49.923357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.287276+0000","last_change":"2024-09-21T03:27:15.287512+0000","last_active":"2024-09-21T03:27:15.287276+0000","last_peered":"2024-09-21T03:27:15.287276+0000","last_clean":"2024-09-21T03:27:15.287276+0000","last_became_active":"2024-09-21T03:27:13.271520+0000","last_became_peered":"2024-09-21T03:27:13.271520+0000","last_unstale":"2024-09-21T03:27:15.287276+0000","last_undegraded":"2024-09-21T03:27:15.287276+0000","last_fullsized":"2024-09-21T03:27:15.287276+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:54.936725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049584300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.286810+0000","last_change":"2024-09-21T03:27:15.282023+0000","last_active":"2024-09-21T03:27:15.286810+0000","last_peered":"2024-09-21T03:27:15.286810+0000","last_clean":"2024-09-21T03:27:15.286810+0000","last_became_active":"2024-09-21T03:27:13.276424+0000","last_became_peered":"2024-09-21T03:27:13.276424+0000","last_unstale":"2024-09-21T03:27:15.286810+0000","last_undegraded":"2024-09-21T03:27:15.286810+0000","last_fullsized":"2024-09-21T03:27:15.286810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:23.923919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010094349999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.273538+0000","last_change":"2024-09-21T03:27:15.273538+0000","last_active":"2024-09-21T03:27:15.273538+0000","last_peered":"2024-09-21T03:27:15.273538+0000","last_clean":"2024-09-21T03:27:15.273538+0000","last_became_active":"2024-09-21T03:27:13.273714+0000","last_became_peered":"2024-09-21T03:27:13.273714+0000","last_unstale":"2024-09-21T03:27:15.273538+0000","last_undegraded":"2024-09-21T03:27:15.273538+0000","last_fullsized":"2024-09-21T03:27:15.273538+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:31:40.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056217599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.281443+0000","last_change":"2024-09-21T03:27:15.281443+0000","last_active":"2024-09-21T03:27:15.281443+0000","last_peered":"2024-09-21T03:27:15.281443+0000","last_clean":"2024-09-21T03:27:15.281443+0000","last_became_active":"2024-09-21T03:27:13.273967+0000","last_became_peered":"2024-09-21T03:27:13.273967+0000","last_unstale":"2024-09-21T03:27:15.281443+0000","last_undegraded":"2024-09-21T03:27:15.281443+0000","last_fullsized":"2024-09-21T03:27:15.281443+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:53.091651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045280299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.282372+0000","last_change":"2024-09-21T03:27:15.282372+0000","last_active":"2024-09-21T03:27:15.282372+0000","last_peered":"2024-09-21T03:27:15.282372+0000","last_clean":"2024-09-21T03:27:15.282372+0000","last_became_active":"2024-09-21T03:27:13.276509+0000","last_became_peered":"2024-09-21T03:27:13.276509+0000","last_unstale":"2024-09-21T03:27:15.282372+0000","last_undegraded":"2024-09-21T03:27:15.282372+0000","last_fullsized":"2024-09-21T03:27:15.282372+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:22:49.703014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013599459999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'97","reported_seq":147,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.286924+0000","last_change":"2024-09-21T03:27:09.247989+0000","last_active":"2024-09-21T03:27:15.286924+0000","last_peered":"2024-09-21T03:27:15.286924+0000","last_clean":"2024-09-21T03:27:15.286924+0000","last_became_active":"2024-09-21T03:27:09.247627+0000","last_became_peered":"2024-09-21T03:27:09.247627+0000","last_unstale":"2024-09-21T03:27:15.286924+0000","last_undegraded":"2024-09-21T03:27:15.286924+0000","last_fullsized":"2024-09-21T03:27:15.286924+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_clean_scrub_stamp":"2024-09-21T03:27:08.233475+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:30:49.675995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":43193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":43174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":749568,"data_stored":633542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31932,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":753664,"data_stored":633561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T03:27:19.893 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T03:27:19.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.995+0000 7fefeb86b640 1 -- 172.21.15.27:0/2265626608 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fefe40988d0 msgr2=0x7fefe4098cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:19.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.995+0000 7fefeb86b640 1 --2- 172.21.15.27:0/2265626608 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fefe40988d0 0x7fefe4098cb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009860 tx=0x7fefe002a4b0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.996+0000 7fefeb86b640 1 -- 172.21.15.27:0/2265626608 shutdown_connections 2024-09-21T03:27:19.997 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.996+0000 7fefeb86b640 1 --2- 172.21.15.27:0/2265626608 >> v2:172.21.15.27:3300/0 conn(0x7fefe40991f0 0x7fefe40963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.996+0000 7fefeb86b640 1 --2- 172.21.15.27:0/2265626608 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fefe40988d0 0x7fefe4098cb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:19.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.996+0000 7fefeb86b640 1 -- 172.21.15.27:0/2265626608 >> 172.21.15.27:0/2265626608 conn(0x7fefe40a0a60 msgr2=0x7fefe409cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:19.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.996+0000 7fefeb86b640 1 -- 172.21.15.27:0/2265626608 shutdown_connections 2024-09-21T03:27:19.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.996+0000 7fefeb86b640 1 -- 172.21.15.27:0/2265626608 wait complete. 2024-09-21T03:27:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefeb86b640 1 Processor -- start 2024-09-21T03:27:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefeb86b640 1 -- start start 2024-09-21T03:27:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefeb86b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fefe4096920 0x7fefe406ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefeb86b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fefe40988d0 0x7fefe40704e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefeb86b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe40991f0 0x7fefe4070a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefeb86b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fefe41369c0 con 0x7fefe40991f0 2024-09-21T03:27:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefeb86b640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fefe4136840 con 0x7fefe40988d0 2024-09-21T03:27:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefeb86b640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fefe4136b40 con 0x7fefe4096920 2024-09-21T03:27:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe9e08640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe40991f0 0x7fefe4070a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe8e06640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fefe40988d0 0x7fefe40704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe9607640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fefe4096920 0x7fefe406ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe8e06640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fefe40988d0 0x7fefe40704e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:56808/0 (socket says 172.21.15.27:56808) 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe8e06640 1 -- 172.21.15.27:0/1240919813 learned_addr learned my addr 172.21.15.27:0/1240919813 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe9e08640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe40991f0 0x7fefe4070a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40992/0 (socket says 172.21.15.27:40992) 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe9e08640 1 -- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fefe4096920 msgr2=0x7fefe406ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe9e08640 1 --2- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fefe4096920 0x7fefe406ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe9e08640 1 -- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fefe40988d0 msgr2=0x7fefe40704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe9e08640 1 --2- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fefe40988d0 0x7fefe40704e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe9e08640 1 -- 172.21.15.27:0/1240919813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe40718a0 con 0x7fefe40991f0 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe9607640 1 --2- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fefe4096920 0x7fefe406ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.997+0000 7fefe8e06640 1 --2- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fefe40988d0 0x7fefe40704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.998+0000 7fefe9e08640 1 --2- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe40991f0 0x7fefe4070a20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00cb10 tx=0x7fefdc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.998+0000 7fefd9ffb640 1 -- 172.21.15.27:0/1240919813 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc016020 con 0x7fefe40991f0 2024-09-21T03:27:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.998+0000 7fefd9ffb640 1 -- 172.21.15.27:0/1240919813 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fefdc00ad00 con 0x7fefe40991f0 2024-09-21T03:27:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.998+0000 7fefd9ffb640 1 -- 172.21.15.27:0/1240919813 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc012680 con 0x7fefe40991f0 2024-09-21T03:27:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.998+0000 7fefeb86b640 1 -- 172.21.15.27:0/1240919813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe4070f60 con 0x7fefe40991f0 2024-09-21T03:27:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.998+0000 7fefeb86b640 1 -- 172.21.15.27:0/1240919813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe41c9d90 con 0x7fefe40991f0 2024-09-21T03:27:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.999+0000 7fefd9ffb640 1 -- 172.21.15.27:0/1240919813 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7fefdc019020 con 0x7fefe40991f0 2024-09-21T03:27:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.999+0000 7fefeb86b640 1 -- 172.21.15.27:0/1240919813 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe4098cb0 con 0x7fefe40991f0 2024-09-21T03:27:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:19.999+0000 7fefd9ffb640 1 -- 172.21.15.27:0/1240919813 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 2840+0+0 (secure 0 0 0) 0x7fefdc05dc90 con 0x7fefe40991f0 2024-09-21T03:27:20.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.003+0000 7fefd9ffb640 1 -- 172.21.15.27:0/1240919813 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7fefdc020030 con 0x7fefe40991f0 2024-09-21T03:27:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.149+0000 7fefeb86b640 1 -- 172.21.15.27:0/1240919813 --> v1:172.21.15.27:6800/1098903051 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fefe4067c80 con 0x7fefc8047fd0 2024-09-21T03:27:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.150+0000 7fefd9ffb640 1 -- 172.21.15.27:0/1240919813 <== mgr.4102 v1:172.21.15.27:6800/1098903051 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+32222 (unknown 2965378022 0 1129619846) 0x7fefe4067c80 con 0x7fefc8047fd0 2024-09-21T03:27:20.152 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:27:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-21T03:27:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.152+0000 7fefeb86b640 1 -- 172.21.15.27:0/1240919813 >> v1:172.21.15.27:6800/1098903051 conn(0x7fefc8047fd0 legacy=0x7fefc804a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.152+0000 7fefeb86b640 1 -- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe40991f0 msgr2=0x7fefe4070a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.152+0000 7fefeb86b640 1 --2- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe40991f0 0x7fefe4070a20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00cb10 tx=0x7fefdc00a060 comp rx=0 tx=0).stop 2024-09-21T03:27:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.153+0000 7fefeb86b640 1 -- 172.21.15.27:0/1240919813 shutdown_connections 2024-09-21T03:27:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.153+0000 7fefeb86b640 1 --2- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fefe40991f0 0x7fefe4070a20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.153+0000 7fefeb86b640 1 --2- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fefe40988d0 0x7fefe40704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.153+0000 7fefeb86b640 1 --2- 172.21.15.27:0/1240919813 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fefe4096920 0x7fefe406ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.153+0000 7fefeb86b640 1 -- 172.21.15.27:0/1240919813 >> 172.21.15.27:0/1240919813 conn(0x7fefe40a0a60 msgr2=0x7fefe4127500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.153+0000 7fefeb86b640 1 -- 172.21.15.27:0/1240919813 shutdown_connections 2024-09-21T03:27:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.153+0000 7fefeb86b640 1 -- 172.21.15.27:0/1240919813 wait complete. 2024-09-21T03:27:20.164 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-21T03:27:19.501994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":114072,"kb_used_data":1776,"kb_used_omap":6,"kb_used_meta":112185,"kb_avail":377373288,"statfs":{"total":386547056640,"available":386430246912,"internally_reserved":0,"allocated":1818624,"data_stored":1353470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":114878231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":16,"num_write_kb":264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000932"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.282474+0000","last_change":"2024-09-21T03:27:15.282474+0000","last_active":"2024-09-21T03:27:15.282474+0000","last_peered":"2024-09-21T03:27:15.282474+0000","last_clean":"2024-09-21T03:27:15.282474+0000","last_became_active":"2024-09-21T03:27:13.273870+0000","last_became_peered":"2024-09-21T03:27:13.273870+0000","last_unstale":"2024-09-21T03:27:15.282474+0000","last_undegraded":"2024-09-21T03:27:15.282474+0000","last_fullsized":"2024-09-21T03:27:15.282474+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:22.300880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393285,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.287209+0000","last_change":"2024-09-21T03:27:15.287459+0000","last_active":"2024-09-21T03:27:15.287209+0000","last_peered":"2024-09-21T03:27:15.287209+0000","last_clean":"2024-09-21T03:27:15.287209+0000","last_became_active":"2024-09-21T03:27:13.271654+0000","last_became_peered":"2024-09-21T03:27:13.271654+0000","last_unstale":"2024-09-21T03:27:15.287209+0000","last_undegraded":"2024-09-21T03:27:15.287209+0000","last_fullsized":"2024-09-21T03:27:15.287209+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:45:52.098322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044937200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.282111+0000","last_change":"2024-09-21T03:27:15.282111+0000","last_active":"2024-09-21T03:27:15.282111+0000","last_peered":"2024-09-21T03:27:15.282111+0000","last_clean":"2024-09-21T03:27:15.282111+0000","last_became_active":"2024-09-21T03:27:13.273857+0000","last_became_peered":"2024-09-21T03:27:13.273857+0000","last_unstale":"2024-09-21T03:27:15.282111+0000","last_undegraded":"2024-09-21T03:27:15.282111+0000","last_fullsized":"2024-09-21T03:27:15.282111+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:47:49.923357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.287276+0000","last_change":"2024-09-21T03:27:15.287512+0000","last_active":"2024-09-21T03:27:15.287276+0000","last_peered":"2024-09-21T03:27:15.287276+0000","last_clean":"2024-09-21T03:27:15.287276+0000","last_became_active":"2024-09-21T03:27:13.271520+0000","last_became_peered":"2024-09-21T03:27:13.271520+0000","last_unstale":"2024-09-21T03:27:15.287276+0000","last_undegraded":"2024-09-21T03:27:15.287276+0000","last_fullsized":"2024-09-21T03:27:15.287276+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:54.936725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049584300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.286810+0000","last_change":"2024-09-21T03:27:15.282023+0000","last_active":"2024-09-21T03:27:15.286810+0000","last_peered":"2024-09-21T03:27:15.286810+0000","last_clean":"2024-09-21T03:27:15.286810+0000","last_became_active":"2024-09-21T03:27:13.276424+0000","last_became_peered":"2024-09-21T03:27:13.276424+0000","last_unstale":"2024-09-21T03:27:15.286810+0000","last_undegraded":"2024-09-21T03:27:15.286810+0000","last_fullsized":"2024-09-21T03:27:15.286810+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:23.923919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010094349999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.273538+0000","last_change":"2024-09-21T03:27:15.273538+0000","last_active":"2024-09-21T03:27:15.273538+0000","last_peered":"2024-09-21T03:27:15.273538+0000","last_clean":"2024-09-21T03:27:15.273538+0000","last_became_active":"2024-09-21T03:27:13.273714+0000","last_became_peered":"2024-09-21T03:27:13.273714+0000","last_unstale":"2024-09-21T03:27:15.273538+0000","last_undegraded":"2024-09-21T03:27:15.273538+0000","last_fullsized":"2024-09-21T03:27:15.273538+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:31:40.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056217599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.281443+0000","last_change":"2024-09-21T03:27:15.281443+0000","last_active":"2024-09-21T03:27:15.281443+0000","last_peered":"2024-09-21T03:27:15.281443+0000","last_clean":"2024-09-21T03:27:15.281443+0000","last_became_active":"2024-09-21T03:27:13.273967+0000","last_became_peered":"2024-09-21T03:27:13.273967+0000","last_unstale":"2024-09-21T03:27:15.281443+0000","last_undegraded":"2024-09-21T03:27:15.281443+0000","last_fullsized":"2024-09-21T03:27:15.281443+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:53.091651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045280299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.282372+0000","last_change":"2024-09-21T03:27:15.282372+0000","last_active":"2024-09-21T03:27:15.282372+0000","last_peered":"2024-09-21T03:27:15.282372+0000","last_clean":"2024-09-21T03:27:15.282372+0000","last_became_active":"2024-09-21T03:27:13.276509+0000","last_became_peered":"2024-09-21T03:27:13.276509+0000","last_unstale":"2024-09-21T03:27:15.282372+0000","last_undegraded":"2024-09-21T03:27:15.282372+0000","last_fullsized":"2024-09-21T03:27:15.282372+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:22:49.703014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013599459999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'97","reported_seq":147,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-21T03:27:15.286924+0000","last_change":"2024-09-21T03:27:09.247989+0000","last_active":"2024-09-21T03:27:15.286924+0000","last_peered":"2024-09-21T03:27:15.286924+0000","last_clean":"2024-09-21T03:27:15.286924+0000","last_became_active":"2024-09-21T03:27:09.247627+0000","last_became_peered":"2024-09-21T03:27:09.247627+0000","last_unstale":"2024-09-21T03:27:15.286924+0000","last_undegraded":"2024-09-21T03:27:15.286924+0000","last_fullsized":"2024-09-21T03:27:15.286924+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_clean_scrub_stamp":"2024-09-21T03:27:08.233475+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:30:49.675995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":43193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":43174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":749568,"data_stored":633542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31932,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339908,"statfs":{"total":96636764160,"available":96604065792,"internally_reserved":0,"allocated":753664,"data_stored":633561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T03:27:20.165 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T03:27:20.165 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T03:27:20.165 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T03:27:20.165 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T03:27:20.268 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.266+0000 7fca3a653640 1 -- 172.21.15.27:0/1383400966 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca3412dbe0 msgr2=0x7fca3412dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.266+0000 7fca3a653640 1 --2- 172.21.15.27:0/1383400966 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca3412dbe0 0x7fca3412dfc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fca24009830 tx=0x7fca2402a3f0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.267+0000 7fca3a653640 1 -- 172.21.15.27:0/1383400966 shutdown_connections 2024-09-21T03:27:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.267+0000 7fca3a653640 1 --2- 172.21.15.27:0/1383400966 >> v2:172.21.15.27:3300/0 conn(0x7fca34132f00 0x7fca341352f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.267+0000 7fca3a653640 1 --2- 172.21.15.27:0/1383400966 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca3412dbe0 0x7fca3412dfc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.267+0000 7fca3a653640 1 -- 172.21.15.27:0/1383400966 >> 172.21.15.27:0/1383400966 conn(0x7fca340a0350 msgr2=0x7fca340a0760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.267+0000 7fca3a653640 1 -- 172.21.15.27:0/1383400966 shutdown_connections 2024-09-21T03:27:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.267+0000 7fca3a653640 1 -- 172.21.15.27:0/1383400966 wait complete. 2024-09-21T03:27:20.269 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.267+0000 7fca3a653640 1 Processor -- start 2024-09-21T03:27:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca3a653640 1 -- start start 2024-09-21T03:27:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca3a653640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca34098f20 0x7fca341c3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca3a653640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca3412dbe0 0x7fca341c4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:20.271 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca3a653640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fca34132f00 0x7fca341c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca3a653640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fca34136610 con 0x7fca34098f20 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca3a653640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fca34136490 con 0x7fca3412dbe0 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca3a653640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fca34136b90 con 0x7fca34132f00 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca33fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca34098f20 0x7fca341c3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca337fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca3412dbe0 0x7fca341c4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca33fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca34098f20 0x7fca341c3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41008/0 (socket says 172.21.15.27:41008) 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca38bf0640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fca34132f00 0x7fca341c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca33fff640 1 -- 172.21.15.27:0/2326362566 learned_addr learned my addr 172.21.15.27:0/2326362566 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca33fff640 1 -- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fca34132f00 msgr2=0x7fca341c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca33fff640 1 --2- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fca34132f00 0x7fca341c4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca33fff640 1 -- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca3412dbe0 msgr2=0x7fca341c4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca33fff640 1 --2- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca3412dbe0 0x7fca341c4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.272 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca33fff640 1 -- 172.21.15.27:0/2326362566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca34131a70 con 0x7fca34098f20 2024-09-21T03:27:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca38bf0640 1 --2- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fca34132f00 0x7fca341c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.268+0000 7fca337fe640 1 --2- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca3412dbe0 0x7fca341c4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.269+0000 7fca33fff640 1 --2- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca34098f20 0x7fca341c3dd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fca24009950 tx=0x7fca24032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.269+0000 7fca30ff9640 1 -- 172.21.15.27:0/2326362566 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24040070 con 0x7fca34098f20 2024-09-21T03:27:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.269+0000 7fca30ff9640 1 -- 172.21.15.27:0/2326362566 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fca24005310 con 0x7fca34098f20 2024-09-21T03:27:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.269+0000 7fca3a653640 1 -- 172.21.15.27:0/2326362566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca341306b0 con 0x7fca34098f20 2024-09-21T03:27:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.269+0000 7fca30ff9640 1 -- 172.21.15.27:0/2326362566 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24005c90 con 0x7fca34098f20 2024-09-21T03:27:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.269+0000 7fca3a653640 1 -- 172.21.15.27:0/2326362566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca3412ecd0 con 0x7fca34098f20 2024-09-21T03:27:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.270+0000 7fca3a653640 1 -- 172.21.15.27:0/2326362566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca3412dfc0 con 0x7fca34098f20 2024-09-21T03:27:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.270+0000 7fca30ff9640 1 -- 172.21.15.27:0/2326362566 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7fca24039040 con 0x7fca34098f20 2024-09-21T03:27:20.273 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.270+0000 7fca30ff9640 1 -- 172.21.15.27:0/2326362566 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 2840+0+0 (secure 0 0 0) 0x7fca2407e890 con 0x7fca34098f20 2024-09-21T03:27:20.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.274+0000 7fca30ff9640 1 -- 172.21.15.27:0/2326362566 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7fca34131a70 con 0x7fca34098f20 2024-09-21T03:27:20.452 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.450+0000 7fca3a653640 1 -- 172.21.15.27:0/2326362566 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fca3409cd60 con 0x7fca34098f20 2024-09-21T03:27:20.453 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.450+0000 7fca30ff9640 1 -- 172.21.15.27:0/2326362566 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fca2407da00 con 0x7fca34098f20 2024-09-21T03:27:20.453 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:27:20.453 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T03:27:20.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.452+0000 7fca3a653640 1 -- 172.21.15.27:0/2326362566 >> v1:172.21.15.27:6800/1098903051 conn(0x7fca20047fd0 legacy=0x7fca2004a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.452+0000 7fca3a653640 1 -- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca34098f20 msgr2=0x7fca341c3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.452+0000 7fca3a653640 1 --2- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca34098f20 0x7fca341c3dd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fca24009950 tx=0x7fca24032c10 comp rx=0 tx=0).stop 2024-09-21T03:27:20.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.452+0000 7fca3a653640 1 -- 172.21.15.27:0/2326362566 shutdown_connections 2024-09-21T03:27:20.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.452+0000 7fca3a653640 1 --2- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fca34132f00 0x7fca341c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.452+0000 7fca3a653640 1 --2- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fca3412dbe0 0x7fca341c4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.454 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.452+0000 7fca3a653640 1 --2- 172.21.15.27:0/2326362566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fca34098f20 0x7fca341c3dd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.452+0000 7fca3a653640 1 -- 172.21.15.27:0/2326362566 >> 172.21.15.27:0/2326362566 conn(0x7fca340a0350 msgr2=0x7fca3409ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:20.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.452+0000 7fca3a653640 1 -- 172.21.15.27:0/2326362566 shutdown_connections 2024-09-21T03:27:20.455 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.452+0000 7fca3a653640 1 -- 172.21.15.27:0/2326362566 wait complete. 2024-09-21T03:27:20.464 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T03:27:20.464 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T03:27:20.475 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T03:27:20.475 INFO:teuthology.task.exec:Running commands on role osd.0 host ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:27:20.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph osd set-require-min-compat-client squid' 2024-09-21T03:27:20.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.575+0000 7f429212c640 1 -- 172.21.15.27:0/2304400722 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f428c12e300 msgr2=0x7f428c135f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.575+0000 7f429212c640 1 --2- 172.21.15.27:0/2304400722 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f428c12e300 0x7f428c135f30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4284005fd0 tx=0x7f428402a780 comp rx=0 tx=0).stop 2024-09-21T03:27:20.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.576+0000 7f429212c640 1 -- 172.21.15.27:0/2304400722 shutdown_connections 2024-09-21T03:27:20.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.576+0000 7f429212c640 1 --2- 172.21.15.27:0/2304400722 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f428c12e300 0x7f428c135f30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.576+0000 7f429212c640 1 --2- 172.21.15.27:0/2304400722 >> v2:172.21.15.27:3300/0 conn(0x7f428c098140 0x7f428c098520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.577 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.576+0000 7f429212c640 1 -- 172.21.15.27:0/2304400722 >> 172.21.15.27:0/2304400722 conn(0x7f428c09e700 msgr2=0x7f428c09eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:20.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.576+0000 7f429212c640 1 -- 172.21.15.27:0/2304400722 shutdown_connections 2024-09-21T03:27:20.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.576+0000 7f429212c640 1 -- 172.21.15.27:0/2304400722 wait complete. 2024-09-21T03:27:20.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.576+0000 7f429212c640 1 Processor -- start 2024-09-21T03:27:20.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f429212c640 1 -- start start 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f429212c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f428c098140 0x7f428c1c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f429212c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c098a60 0x7f428c1c4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f429212c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f428c12e300 0x7f428c070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f429212c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f428c136a40 con 0x7f428c098a60 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f429212c640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f428c1368c0 con 0x7f428c098140 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f429212c640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f428c136bc0 con 0x7f428c12e300 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f428b7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f428c098140 0x7f428c1c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f428affd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c098a60 0x7f428c1c4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f428b7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f428c098140 0x7f428c1c4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:56852/0 (socket says 172.21.15.27:56852) 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f428b7fe640 1 -- 172.21.15.27:0/3702347576 learned_addr learned my addr 172.21.15.27:0/3702347576 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f428bfff640 1 --2- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f428c12e300 0x7f428c070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f428affd640 1 -- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f428c12e300 msgr2=0x7f428c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.580 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f428affd640 1 --2- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f428c12e300 0x7f428c070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f428affd640 1 -- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f428c098140 msgr2=0x7f428c1c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:20.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f428affd640 1 --2- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f428c098140 0x7f428c1c4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:20.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f428affd640 1 -- 172.21.15.27:0/3702347576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428c072fa0 con 0x7f428c098a60 2024-09-21T03:27:20.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.577+0000 7f428b7fe640 1 --2- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f428c098140 0x7f428c1c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:20.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.578+0000 7f428affd640 1 --2- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c098a60 0x7f428c1c4640 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f428000c370 tx=0x7f428000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:20.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.578+0000 7f4273fff640 1 -- 172.21.15.27:0/3702347576 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4280011790 con 0x7f428c098a60 2024-09-21T03:27:20.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.578+0000 7f4273fff640 1 -- 172.21.15.27:0/3702347576 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f4280017070 con 0x7f428c098a60 2024-09-21T03:27:20.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.578+0000 7f4273fff640 1 -- 172.21.15.27:0/3702347576 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4280011cc0 con 0x7f428c098a60 2024-09-21T03:27:20.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.578+0000 7f429212c640 1 -- 172.21.15.27:0/3702347576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428c128710 con 0x7f428c098a60 2024-09-21T03:27:20.581 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.578+0000 7f429212c640 1 -- 172.21.15.27:0/3702347576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f428c1c4b80 con 0x7f428c098a60 2024-09-21T03:27:20.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.579+0000 7f4273fff640 1 -- 172.21.15.27:0/3702347576 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f428001c460 con 0x7f428c098a60 2024-09-21T03:27:20.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.579+0000 7f429212c640 1 -- 172.21.15.27:0/3702347576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f428c098520 con 0x7f428c098a60 2024-09-21T03:27:20.582 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.579+0000 7f4273fff640 1 -- 172.21.15.27:0/3702347576 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 2840+0+0 (secure 0 0 0) 0x7f428005d6d0 con 0x7f428c098a60 2024-09-21T03:27:20.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.583+0000 7f4273fff640 1 -- 172.21.15.27:0/3702347576 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7f428c072fa0 con 0x7f428c098a60 2024-09-21T03:27:20.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:20.740+0000 7f429212c640 1 -- 172.21.15.27:0/3702347576 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd set-require-min-compat-client", "version": "squid"} v 0) -- 0x7f428c06cc20 con 0x7f428c098a60 2024-09-21T03:27:21.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.295+0000 7f4273fff640 1 -- 172.21.15.27:0/3702347576 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd set-require-min-compat-client", "version": "squid"}]=0 set require_min_compat_client to squid v20) ==== 139+0+0 (secure 0 0 0) 0x7f4280011e60 con 0x7f428c098a60 2024-09-21T03:27:21.296 INFO:teuthology.orchestra.run.smithi027.stderr:set require_min_compat_client to squid 2024-09-21T03:27:21.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.297+0000 7f429212c640 1 -- 172.21.15.27:0/3702347576 >> v1:172.21.15.27:6800/1098903051 conn(0x7f4278047fd0 legacy=0x7f427804a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:21.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.297+0000 7f429212c640 1 -- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c098a60 msgr2=0x7f428c1c4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:21.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.297+0000 7f429212c640 1 --2- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c098a60 0x7f428c1c4640 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f428000c370 tx=0x7f428000c830 comp rx=0 tx=0).stop 2024-09-21T03:27:21.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.297+0000 7f429212c640 1 -- 172.21.15.27:0/3702347576 shutdown_connections 2024-09-21T03:27:21.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.297+0000 7f429212c640 1 --2- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f428c12e300 0x7f428c070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:21.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.297+0000 7f429212c640 1 --2- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f428c098a60 0x7f428c1c4640 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:21.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.297+0000 7f429212c640 1 --2- 172.21.15.27:0/3702347576 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f428c098140 0x7f428c1c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:21.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.297+0000 7f429212c640 1 -- 172.21.15.27:0/3702347576 >> 172.21.15.27:0/3702347576 conn(0x7f428c09e700 msgr2=0x7f428c1276f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:21.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.298+0000 7f429212c640 1 -- 172.21.15.27:0/3702347576 shutdown_connections 2024-09-21T03:27:21.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.298+0000 7f429212c640 1 -- 172.21.15.27:0/3702347576 wait complete. 2024-09-21T03:27:21.311 INFO:teuthology.run_tasks:Running task print... 2024-09-21T03:27:21.319 INFO:teuthology.task.print:**** done ceph 2024-09-21T03:27:21.319 INFO:teuthology.run_tasks:Running task print... 2024-09-21T03:27:21.326 INFO:teuthology.task.print:*** upgrading, no cephfs present 2024-09-21T03:27:21.326 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T03:27:21.335 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T03:27:21.336 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:27:21.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs dump' 2024-09-21T03:27:21.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.439+0000 7fdb3e696640 1 -- 172.21.15.27:0/3669081737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb38070970 msgr2=0x7fdb38070dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:21.440 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.439+0000 7fdb3e696640 1 --2- 172.21.15.27:0/3669081737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb38070970 0x7fdb38070dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c009830 tx=0x7fdb2c02a3f0 comp rx=0 tx=0).stop 2024-09-21T03:27:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.439+0000 7fdb3e696640 1 -- 172.21.15.27:0/3669081737 shutdown_connections 2024-09-21T03:27:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.439+0000 7fdb3e696640 1 --2- 172.21.15.27:0/3669081737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb38070970 0x7fdb38070dd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.439+0000 7fdb3e696640 1 --2- 172.21.15.27:0/3669081737 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdb3812bec0 0x7fdb3812c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.439+0000 7fdb3e696640 1 -- 172.21.15.27:0/3669081737 >> 172.21.15.27:0/3669081737 conn(0x7fdb3809c3d0 msgr2=0x7fdb3809c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.439+0000 7fdb3e696640 1 -- 172.21.15.27:0/3669081737 shutdown_connections 2024-09-21T03:27:21.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.439+0000 7fdb3e696640 1 -- 172.21.15.27:0/3669081737 wait complete. 2024-09-21T03:27:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.439+0000 7fdb3e696640 1 Processor -- start 2024-09-21T03:27:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb3e696640 1 -- start start 2024-09-21T03:27:21.443 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb3e696640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fdb38070970 0x7fdb3812fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb3e696640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb38071310 0x7fdb3812ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb3e696640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdb3812bec0 0x7fdb381304d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb3e696640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdb38134810 con 0x7fdb38071310 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb3e696640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fdb38134690 con 0x7fdb3812bec0 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb3e696640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fdb38134990 con 0x7fdb38070970 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb38071310 0x7fdb3812ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb37fff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fdb38070970 0x7fdb3812fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb3cc33640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdb3812bec0 0x7fdb381304d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb377fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb38071310 0x7fdb3812ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:41040/0 (socket says 172.21.15.27:41040) 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.440+0000 7fdb377fe640 1 -- 172.21.15.27:0/3967331755 learned_addr learned my addr 172.21.15.27:0/3967331755 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb3cc33640 1 -- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fdb38070970 msgr2=0x7fdb3812fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:21.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb3cc33640 1 --2- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fdb38070970 0x7fdb3812fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb3cc33640 1 -- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb38071310 msgr2=0x7fdb3812ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb3cc33640 1 --2- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb38071310 0x7fdb3812ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb3cc33640 1 -- 172.21.15.27:0/3967331755 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb381260b0 con 0x7fdb3812bec0 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb377fe640 1 --2- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb38071310 0x7fdb3812ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb37fff640 1 --2- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fdb38070970 0x7fdb3812fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb3cc33640 1 --2- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdb3812bec0 0x7fdb381304d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000c750 tx=0x7fdb3000cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb34ff9640 1 -- 172.21.15.27:0/3967331755 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb30017070 con 0x7fdb3812bec0 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb34ff9640 1 -- 172.21.15.27:0/3967331755 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdb30003cb0 con 0x7fdb3812bec0 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb34ff9640 1 -- 172.21.15.27:0/3967331755 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb300052d0 con 0x7fdb3812bec0 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb3e696640 1 -- 172.21.15.27:0/3967331755 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb38130a10 con 0x7fdb3812bec0 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.441+0000 7fdb3e696640 1 -- 172.21.15.27:0/3967331755 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb38132290 con 0x7fdb3812bec0 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.442+0000 7fdb3e696640 1 -- 172.21.15.27:0/3967331755 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb38070dd0 con 0x7fdb3812bec0 2024-09-21T03:27:21.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.442+0000 7fdb34ff9640 1 -- 172.21.15.27:0/3967331755 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7fdb3001a070 con 0x7fdb3812bec0 2024-09-21T03:27:21.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.443+0000 7fdb34ff9640 1 -- 172.21.15.27:0/3967331755 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7fdb3005cf50 con 0x7fdb3812bec0 2024-09-21T03:27:21.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.446+0000 7fdb34ff9640 1 -- 172.21.15.27:0/3967331755 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+151260 (secure 0 0 0) 0x7fdb380717a0 con 0x7fdb3812bec0 2024-09-21T03:27:21.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.620+0000 7fdb3e696640 1 -- 172.21.15.27:0/3967331755 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fdb38127990 con 0x7fdb3812bec0 2024-09-21T03:27:21.622 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.621+0000 7fdb34ff9640 1 -- 172.21.15.27:0/3967331755 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 1 v1) ==== 75+0+440 (secure 0 0 0) 0x7fdb3005c0e0 con 0x7fdb3812bec0 2024-09-21T03:27:21.622 INFO:teuthology.orchestra.run.smithi027.stdout:e1 2024-09-21T03:27:21.622 INFO:teuthology.orchestra.run.smithi027.stdout:btime 2024-09-21T03:26:54:150286+0000 2024-09-21T03:27:21.622 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T03:27:21.623 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T03:27:21.623 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: -1 2024-09-21T03:27:21.623 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:27:21.623 INFO:teuthology.orchestra.run.smithi027.stdout:No filesystems configured 2024-09-21T03:27:21.623 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 1 2024-09-21T03:27:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.622+0000 7fdb3e696640 1 -- 172.21.15.27:0/3967331755 >> v1:172.21.15.27:6800/1098903051 conn(0x7fdb24047fd0 legacy=0x7fdb2404a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.622+0000 7fdb3e696640 1 -- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdb3812bec0 msgr2=0x7fdb381304d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:27:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.622+0000 7fdb3e696640 1 --2- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdb3812bec0 0x7fdb381304d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000c750 tx=0x7fdb3000cc10 comp rx=0 tx=0).stop 2024-09-21T03:27:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.623+0000 7fdb3e696640 1 -- 172.21.15.27:0/3967331755 shutdown_connections 2024-09-21T03:27:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.623+0000 7fdb3e696640 1 --2- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdb3812bec0 0x7fdb381304d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:21.624 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.623+0000 7fdb3e696640 1 --2- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb38071310 0x7fdb3812ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.623+0000 7fdb3e696640 1 --2- 172.21.15.27:0/3967331755 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fdb38070970 0x7fdb3812fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:27:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.623+0000 7fdb3e696640 1 -- 172.21.15.27:0/3967331755 >> 172.21.15.27:0/3967331755 conn(0x7fdb3809c3d0 msgr2=0x7fdb3809ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:27:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.623+0000 7fdb3e696640 1 -- 172.21.15.27:0/3967331755 shutdown_connections 2024-09-21T03:27:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:27:21.623+0000 7fdb3e696640 1 -- 172.21.15.27:0/3967331755 wait complete. 2024-09-21T03:27:21.635 INFO:teuthology.run_tasks:Running task install.upgrade... 2024-09-21T03:27:21.644 INFO:teuthology.task.install:project ceph config {'mon.a': None} overrides {'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'} 2024-09-21T03:27:21.644 INFO:teuthology.task.install:remote ubuntu@smithi027.front.sepia.ceph.com config {'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'} 2024-09-21T03:27:21.644 INFO:teuthology.task.install:extra packages: [] 2024-09-21T03:27:21.649 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T03:27:21.649 INFO:teuthology.task.install:Upgrading ceph rpm packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd 2024-09-21T03:27:21.649 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph-common --qf '%{VERSION}-%{RELEASE}' 2024-09-21T03:27:21.671 INFO:teuthology.orchestra.run.smithi027.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-21T03:27:21.671 INFO:teuthology.packaging:The installed version of ceph-common is 19.1.1-373.g4e45e6b4.el9 2024-09-21T03:27:21.671 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T03:27:21.831 INFO:teuthology.task.install:Ceph rpm upgrade from 19.1.1-373.g4e45e6b4.el9 to 19.3.0-5120.g18432218 2024-09-21T03:27:21.832 INFO:teuthology.task.install.rpm:Host smithi027 is: centos 9.stream x86_64 2024-09-21T03:27:21.832 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T03:27:21.979 INFO:teuthology.task.install.rpm:Repo base URL: https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T03:27:21.979 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f /etc/yum.repos.d/ceph.repo 2024-09-21T03:27:22.121 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T03:27:22.121 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2024-09-21T03:27:22.121 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T03:27:22.147 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T03:27:22.214 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T03:27:22.280 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T03:27:22.335 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2024-09-21T03:27:22.347 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2024-09-21T03:27:22.590 INFO:teuthology.orchestra.run.smithi027.stdout:60 files removed 2024-09-21T03:27:22.618 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd 2024-09-21T03:27:23.208 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 300 kB/s | 84 kB 00:00 2024-09-21T03:27:23.509 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 95 kB/s | 24 kB 00:00 2024-09-21T03:27:23.784 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-21T03:27:25.501 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.3 MB 00:01 2024-09-21T03:27:30.919 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 20 MB 00:04 2024-09-21T03:27:37.103 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.5 MB 00:02 2024-09-21T03:27:38.662 INFO:teuthology.orchestra.run.smithi027.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 19 kB 00:00 2024-09-21T03:27:39.583 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-21T03:27:45.214 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-21T03:27:47.425 INFO:teuthology.orchestra.run.smithi027.stderr:Package cephadm available, but not installed. 2024-09-21T03:27:47.426 INFO:teuthology.orchestra.run.smithi027.stdout:No match for argument: cephadm 2024-09-21T03:27:47.432 INFO:teuthology.orchestra.run.smithi027.stderr:Package ceph-mgr-dashboard available, but not installed. 2024-09-21T03:27:47.433 INFO:teuthology.orchestra.run.smithi027.stdout:No match for argument: ceph-mgr-dashboard 2024-09-21T03:27:47.433 INFO:teuthology.orchestra.run.smithi027.stderr:Package ceph-mgr-diskprediction-local available, but not installed. 2024-09-21T03:27:47.433 INFO:teuthology.orchestra.run.smithi027.stdout:No match for argument: ceph-mgr-diskprediction-local 2024-09-21T03:27:47.433 INFO:teuthology.orchestra.run.smithi027.stderr:Package ceph-mgr-rook available, but not installed. 2024-09-21T03:27:47.434 INFO:teuthology.orchestra.run.smithi027.stdout:No match for argument: ceph-mgr-rook 2024-09-21T03:27:47.434 INFO:teuthology.orchestra.run.smithi027.stderr:Package ceph-mgr-cephadm available, but not installed. 2024-09-21T03:27:47.434 INFO:teuthology.orchestra.run.smithi027.stdout:No match for argument: ceph-mgr-cephadm 2024-09-21T03:27:47.436 INFO:teuthology.orchestra.run.smithi027.stderr:Package ceph-volume available, but not installed. 2024-09-21T03:27:47.436 INFO:teuthology.orchestra.run.smithi027.stdout:No match for argument: ceph-volume 2024-09-21T03:27:47.531 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2024-09-21T03:27:47.533 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:27:47.533 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2024-09-21T03:27:47.534 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:27:47.534 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2024-09-21T03:27:47.534 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T03:27:47.534 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T03:27:47.534 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T03:27:47.534 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T03:27:47.534 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache 2024-09-21T03:27:47.534 INFO:teuthology.orchestra.run.smithi027.stdout: x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T03:27:47.534 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T03:27:47.534 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T03:27:47.534 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T03:27:47.535 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:27:47.536 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2024-09-21T03:27:47.537 INFO:teuthology.orchestra.run.smithi027.stdout:================================================================================ 2024-09-21T03:27:47.537 INFO:teuthology.orchestra.run.smithi027.stdout:Install 1 Package 2024-09-21T03:27:47.537 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 30 Packages 2024-09-21T03:27:47.537 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:27:47.537 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 122 M 2024-09-21T03:27:47.537 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2024-09-21T03:27:47.766 INFO:teuthology.orchestra.run.smithi027.stdout:(1/31): ceph-19.3.0-5120.g18432218.el9.x86_64.r 51 kB/s | 6.6 kB 00:00 2024-09-21T03:27:47.791 INFO:teuthology.orchestra.run.smithi027.stdout:(2/31): libnbd-1.20.2-2.el9.x86_64.rpm 1.1 MB/s | 171 kB 00:00 2024-09-21T03:27:48.058 INFO:teuthology.orchestra.run.smithi027.stdout:(3/31): ceph-base-19.3.0-5120.g18432218.el9.x86 13 MB/s | 5.5 MB 00:00 2024-09-21T03:27:48.108 INFO:teuthology.orchestra.run.smithi027.stdout:(4/31): ceph-immutable-object-cache-19.3.0-5120 2.9 MB/s | 148 kB 00:00 2024-09-21T03:27:48.242 INFO:teuthology.orchestra.run.smithi027.stdout:(5/31): ceph-mds-19.3.0-5120.g18432218.el9.x86_ 17 MB/s | 2.3 MB 00:00 2024-09-21T03:27:48.267 INFO:teuthology.orchestra.run.smithi027.stdout:(6/31): ceph-fuse-19.3.0-5120.g18432218.el9.x86 1.8 MB/s | 873 kB 00:00 2024-09-21T03:27:48.326 INFO:teuthology.orchestra.run.smithi027.stdout:(7/31): ceph-mgr-19.3.0-5120.g18432218.el9.x86_ 18 MB/s | 1.5 MB 00:00 2024-09-21T03:27:49.151 INFO:teuthology.orchestra.run.smithi027.stdout:(8/31): ceph-common-19.3.0-5120.g18432218.el9.x 14 MB/s | 20 MB 00:01 2024-09-21T03:27:49.335 INFO:teuthology.orchestra.run.smithi027.stdout:(9/31): ceph-osd-19.3.0-5120.g18432218.el9.x86_ 18 MB/s | 18 MB 00:01 2024-09-21T03:27:49.377 INFO:teuthology.orchestra.run.smithi027.stdout:(10/31): ceph-selinux-19.3.0-5120.g18432218.el9 609 kB/s | 25 kB 00:00 2024-09-21T03:27:49.586 INFO:teuthology.orchestra.run.smithi027.stdout:(11/31): ceph-mon-19.3.0-5120.g18432218.el9.x86 3.8 MB/s | 5.0 MB 00:01 2024-09-21T03:27:49.711 INFO:teuthology.orchestra.run.smithi027.stdout:(12/31): ceph-radosgw-19.3.0-5120.g18432218.el9 19 MB/s | 10 MB 00:00 2024-09-21T03:27:49.736 INFO:teuthology.orchestra.run.smithi027.stdout:(13/31): libcephfs-devel-19.3.0-5120.g18432218. 211 kB/s | 32 kB 00:00 2024-09-21T03:27:49.770 INFO:teuthology.orchestra.run.smithi027.stdout:(14/31): libcephfs2-19.3.0-5120.g18432218.el9.x 13 MB/s | 766 kB 00:00 2024-09-21T03:27:49.795 INFO:teuthology.orchestra.run.smithi027.stdout:(15/31): libcephsqlite-19.3.0-5120.g18432218.el 2.6 MB/s | 158 kB 00:00 2024-09-21T03:27:49.820 INFO:teuthology.orchestra.run.smithi027.stdout:(16/31): librados-devel-19.3.0-5120.g18432218.e 2.5 MB/s | 127 kB 00:00 2024-09-21T03:27:49.879 INFO:teuthology.orchestra.run.smithi027.stdout:(17/31): libradosstriper1-19.3.0-5120.g18432218 8.5 MB/s | 507 kB 00:00 2024-09-21T03:27:50.129 INFO:teuthology.orchestra.run.smithi027.stdout:(18/31): librbd1-19.3.0-5120.g18432218.el9.x86_ 13 MB/s | 3.2 MB 00:00 2024-09-21T03:27:50.246 INFO:teuthology.orchestra.run.smithi027.stdout:(19/31): librados2-19.3.0-5120.g18432218.el9.x8 7.5 MB/s | 3.4 MB 00:00 2024-09-21T03:27:50.322 INFO:teuthology.orchestra.run.smithi027.stdout:(20/31): python3-ceph-argparse-19.3.0-5120.g184 601 kB/s | 45 kB 00:00 2024-09-21T03:27:50.431 INFO:teuthology.orchestra.run.smithi027.stdout:(21/31): librgw2-19.3.0-5120.g18432218.el9.x86_ 18 MB/s | 5.3 MB 00:00 2024-09-21T03:27:50.455 INFO:teuthology.orchestra.run.smithi027.stdout:(22/31): python3-ceph-common-19.3.0-5120.g18432 1.1 MB/s | 148 kB 00:00 2024-09-21T03:27:50.489 INFO:teuthology.orchestra.run.smithi027.stdout:(23/31): python3-cephfs-19.3.0-5120.g18432218.e 2.7 MB/s | 161 kB 00:00 2024-09-21T03:27:50.523 INFO:teuthology.orchestra.run.smithi027.stdout:(24/31): python3-rados-19.3.0-5120.g18432218.el 4.7 MB/s | 321 kB 00:00 2024-09-21T03:27:50.549 INFO:teuthology.orchestra.run.smithi027.stdout:(25/31): python3-rbd-19.3.0-5120.g18432218.el9. 5.1 MB/s | 307 kB 00:00 2024-09-21T03:27:50.573 INFO:teuthology.orchestra.run.smithi027.stdout:(26/31): python3-rgw-19.3.0-5120.g18432218.el9. 1.9 MB/s | 99 kB 00:00 2024-09-21T03:27:50.599 INFO:teuthology.orchestra.run.smithi027.stdout:(27/31): rbd-fuse-19.3.0-5120.g18432218.el9.x86 1.7 MB/s | 86 kB 00:00 2024-09-21T03:27:50.649 INFO:teuthology.orchestra.run.smithi027.stdout:(28/31): rbd-nbd-19.3.0-5120.g18432218.el9.x86_ 3.3 MB/s | 170 kB 00:00 2024-09-21T03:27:50.725 INFO:teuthology.orchestra.run.smithi027.stdout:(29/31): ceph-mgr-modules-core-19.3.0-5120.g184 3.8 MB/s | 292 kB 00:00 2024-09-21T03:27:51.150 INFO:teuthology.orchestra.run.smithi027.stdout:(30/31): ceph-test-19.3.0-5120.g18432218.el9.x8 22 MB/s | 40 MB 00:01 2024-09-21T03:27:51.208 INFO:teuthology.orchestra.run.smithi027.stdout:(31/31): rbd-mirror-19.3.0-5120.g18432218.el9.x 4.9 MB/s | 3.1 MB 00:00 2024-09-21T03:27:51.212 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2024-09-21T03:27:51.212 INFO:teuthology.orchestra.run.smithi027.stdout:Total 33 MB/s | 122 MB 00:03 2024-09-21T03:27:51.228 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2024-09-21T03:27:51.291 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2024-09-21T03:27:51.291 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2024-09-21T03:27:52.153 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2024-09-21T03:27:52.153 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2024-09-21T03:27:54.871 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2024-09-21T03:27:54.985 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 1/61 2024-09-21T03:27:55.097 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 1/61 2024-09-21T03:27:55.152 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2/61 2024-09-21T03:27:55.784 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2/61 2024-09-21T03:27:55.878 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 3/61 2024-09-21T03:27:55.972 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 3/61 2024-09-21T03:27:56.039 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 4/61 2024-09-21T03:27:56.455 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 5/61 2024-09-21T03:27:56.523 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 6/61 2024-09-21T03:27:56.599 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 6/61 2024-09-21T03:27:56.633 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 7/61 2024-09-21T03:27:56.741 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 7/61 2024-09-21T03:27:56.784 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_6 8/61 2024-09-21T03:27:56.851 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_6 8/61 2024-09-21T03:27:56.903 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : python3-ceph-argparse-2:19.3.0-5120.g18432218.el9. 9/61 2024-09-21T03:27:56.976 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 10/61 2024-09-21T03:27:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 11/61 2024-09-21T03:27:57.084 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 12/61 2024-09-21T03:27:57.248 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 13/61 2024-09-21T03:27:57.369 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9. 14/61 2024-09-21T03:27:57.416 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : python3-ceph-common-2:19.3.0-5120.g18432218.el9.x8 15/61 2024-09-21T03:27:59.977 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 16/61 2024-09-21T03:28:00.067 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 16/61 2024-09-21T03:28:00.790 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 16/61 2024-09-21T03:28:00.988 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 17/61 2024-09-21T03:28:01.052 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 17/61 2024-09-21T03:28:01.089 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 18/61 2024-09-21T03:28:10.019 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 18/61 2024-09-21T03:28:10.103 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 19/61 2024-09-21T03:28:10.253 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 19/61 2024-09-21T03:28:10.362 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 20/61 2024-09-21T03:28:10.943 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 20/61 2024-09-21T03:28:10.998 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 21/61 2024-09-21T03:28:13.004 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 21/61 2024-09-21T03:28:13.041 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 22/61 2024-09-21T03:28:13.054 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 22/61 2024-09-21T03:28:14.355 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-2:19.3.0-5120.g18432218.el9.x86_64 23/61 2024-09-21T03:28:14.401 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 24/61 2024-09-21T03:28:14.425 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 24/61 2024-09-21T03:28:14.469 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-immutable-object-cache-2:19.3.0-5120.g1843221 25/61 2024-09-21T03:28:14.869 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g1843221 25/61 2024-09-21T03:28:14.919 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 26/61 2024-09-21T03:28:20.546 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 26/61 2024-09-21T03:28:20.684 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 27/61 2024-09-21T03:28:20.749 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 28/61 2024-09-21T03:28:20.805 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 29/61 2024-09-21T03:28:20.929 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 30/61 2024-09-21T03:28:20.990 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 31/61 2024-09-21T03:28:21.023 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/61 2024-09-21T03:28:21.060 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/61 2024-09-21T03:28:21.066 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 34/61 2024-09-21T03:28:21.066 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 34/61 2024-09-21T03:28:21.103 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 34/61 2024-09-21T03:28:21.128 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/61 2024-09-21T03:28:21.133 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/61 2024-09-21T03:28:21.133 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/61 2024-09-21T03:28:21.162 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/61 2024-09-21T03:28:21.167 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/61 2024-09-21T03:28:21.167 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/61 2024-09-21T03:28:21.195 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/61 2024-09-21T03:28:21.221 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/61 2024-09-21T03:28:21.227 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4 39/61 2024-09-21T03:28:21.227 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4 39/61 2024-09-21T03:28:21.263 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4 39/61 2024-09-21T03:28:21.288 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 40/61 2024-09-21T03:28:21.289 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 41/61 2024-09-21T03:28:21.373 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 41/61 2024-09-21T03:28:21.397 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 42/61 2024-09-21T03:28:21.402 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 43/61 2024-09-21T03:28:21.402 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 43/61 2024-09-21T03:28:21.431 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 43/61 2024-09-21T03:28:21.436 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 44/61 2024-09-21T03:28:21.436 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 44/61 2024-09-21T03:28:21.465 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 44/61 2024-09-21T03:28:21.470 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 45/61 2024-09-21T03:28:21.470 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 45/61 2024-09-21T03:28:21.507 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 45/61 2024-09-21T03:28:21.508 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 46/61 2024-09-21T03:28:21.549 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 46/61 2024-09-21T03:28:21.557 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 47/61 2024-09-21T03:28:21.557 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 47/61 2024-09-21T03:28:21.687 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 47/61 2024-09-21T03:28:21.687 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 48/61 2024-09-21T03:28:21.720 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 48/61 2024-09-21T03:28:21.720 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 49/61 2024-09-21T03:28:21.804 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 49/61 2024-09-21T03:28:21.829 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 50/61 2024-09-21T03:28:21.863 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 51/61 2024-09-21T03:28:21.863 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 52/61 2024-09-21T03:28:21.956 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 52/61 2024-09-21T03:28:21.981 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 53/61 2024-09-21T03:28:21.981 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 54/61 2024-09-21T03:28:22.066 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 54/61 2024-09-21T03:28:22.091 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 55/61 2024-09-21T03:28:22.091 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 56/61 2024-09-21T03:28:22.175 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 56/61 2024-09-21T03:28:22.200 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 57/61 2024-09-21T03:28:22.226 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 58/61 2024-09-21T03:28:22.251 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 59/61 2024-09-21T03:28:22.285 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 60/61 2024-09-21T03:28:22.285 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 61/61 2024-09-21T03:28:23.435 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 61/61 2024-09-21T03:28:23.435 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 1/61 2024-09-21T03:28:23.435 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 2/61 2024-09-21T03:28:23.435 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 4/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 6/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 8/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 9/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g1843221 10/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4 11/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 12/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 14/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 15/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 16/61 2024-09-21T03:28:23.436 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 17/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 18/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 19/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 20/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 21/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 22/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 24/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 25/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 26/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 28/61 2024-09-21T03:28:23.437 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/61 2024-09-21T03:28:23.441 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 30/61 2024-09-21T03:28:23.441 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/61 2024-09-21T03:28:23.441 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 32/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 34/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_6 36/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 38/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 39/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 40/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 41/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el9. 42/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 43/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9.x8 44/61 2024-09-21T03:28:23.442 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 45/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 46/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 47/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 48/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 49/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 50/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 51/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 52/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 53/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 54/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 55/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 56/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 57/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 58/61 2024-09-21T03:28:23.443 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 59/61 2024-09-21T03:28:23.444 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9. 60/61 2024-09-21T03:28:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 61/61 2024-09-21T03:28:24.614 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:24.614 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.615 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.616 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T03:28:24.617 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2024-09-21T03:28:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T03:28:24.617 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:24.617 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2024-09-21T03:28:25.020 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T03:28:25.163 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T03:28:25.186 INFO:teuthology.orchestra.run.smithi027.stdout:19.3.0-5120.g18432218.el9 2024-09-21T03:28:25.186 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T03:28:25.186 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T03:28:25.188 INFO:teuthology.run_tasks:Running task print... 2024-09-21T03:28:25.200 INFO:teuthology.task.print:**** done install.upgrade 2024-09-21T03:28:25.200 INFO:teuthology.run_tasks:Running task ceph.restart... 2024-09-21T03:28:25.214 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-21T03:28:25.214 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:28:25.236 INFO:tasks.ceph.mon.a:Stopped 2024-09-21T03:28:25.236 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-21T03:28:25.236 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-21T03:28:25.278 INFO:tasks.ceph.mon.a:Started 2024-09-21T03:28:25.278 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-21T03:28:25.278 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:28:25.292 INFO:tasks.ceph.mon.b:Stopped 2024-09-21T03:28:25.292 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-21T03:28:25.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-21T03:28:25.334 INFO:tasks.ceph.mon.b:Started 2024-09-21T03:28:25.335 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-21T03:28:25.335 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:28:25.347 INFO:tasks.ceph.mon.c:Stopped 2024-09-21T03:28:25.348 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-21T03:28:25.348 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-21T03:28:25.391 INFO:tasks.ceph.mon.c:Started 2024-09-21T03:28:25.391 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-21T03:28:25.392 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:28:25.414 INFO:tasks.ceph.mgr.x:Stopped 2024-09-21T03:28:25.414 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-21T03:28:25.414 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-21T03:28:25.459 INFO:tasks.ceph.mgr.x:Started 2024-09-21T03:28:25.460 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-21T03:28:25.460 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:28:25.489 INFO:tasks.ceph.mgr.y:Stopped 2024-09-21T03:28:25.489 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-21T03:28:25.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-21T03:28:25.532 INFO:tasks.ceph.mgr.y:Started 2024-09-21T03:28:25.532 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-21T03:28:25.541 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T03:28:25.541 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T03:28:25.541 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T03:28:25.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.673+0000 7fe10bb34640 1 -- 172.21.15.27:0/3838455117 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe1040718e0 msgr2=0x7fe10412c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:25.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.673+0000 7fe10bb34640 1 --2- 172.21.15.27:0/3838455117 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe1040718e0 0x7fe10412c4c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8005fd0 tx=0x7fe0f802a540 comp rx=0 tx=0).stop 2024-09-21T03:28:25.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.674+0000 7fe10bb34640 1 -- 172.21.15.27:0/3838455117 shutdown_connections 2024-09-21T03:28:25.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.674+0000 7fe10bb34640 1 --2- 172.21.15.27:0/3838455117 >> v2:172.21.15.27:3300/0 conn(0x7fe10412ca00 0x7fe10412f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:25.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.674+0000 7fe10bb34640 1 --2- 172.21.15.27:0/3838455117 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe1040718e0 0x7fe10412c4c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:25.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.674+0000 7fe10bb34640 1 -- 172.21.15.27:0/3838455117 >> 172.21.15.27:0/3838455117 conn(0x7fe1040a1d70 msgr2=0x7fe1040a2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:25.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.674+0000 7fe10bb34640 1 -- 172.21.15.27:0/3838455117 shutdown_connections 2024-09-21T03:28:25.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.674+0000 7fe10bb34640 1 -- 172.21.15.27:0/3838455117 wait complete. 2024-09-21T03:28:25.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.675+0000 7fe10bb34640 1 Processor -- start 2024-09-21T03:28:25.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.675+0000 7fe10bb34640 1 -- start start 2024-09-21T03:28:25.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe10bb34640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe1040718e0 0x7fe1041c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:25.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe10bb34640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe10412ca00 0x7fe1041c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:25.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe10bb34640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe10412fb30 0x7fe1041c6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:25.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe10bb34640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe1041386e0 con 0x7fe10412fb30 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe10bb34640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fe104138560 con 0x7fe10412ca00 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe10bb34640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fe104138860 con 0x7fe1040718e0 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe1098d0640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe1040718e0 0x7fe1041c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe1090cf640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe10412ca00 0x7fe1041c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe10a0d1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe10412fb30 0x7fe1041c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe1098d0640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe1040718e0 0x7fe1041c5eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60308/0 (socket says 172.21.15.27:60308) 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe1098d0640 1 -- 172.21.15.27:0/1838043585 learned_addr learned my addr 172.21.15.27:0/1838043585 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe1090cf640 1 -- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe1040718e0 msgr2=0x7fe1041c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe1090cf640 1 --2- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe1040718e0 0x7fe1041c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe1090cf640 1 -- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe10412fb30 msgr2=0x7fe1041c6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe1090cf640 1 --2- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe10412fb30 0x7fe1041c6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe1090cf640 1 -- 172.21.15.27:0/1838043585 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe104134640 con 0x7fe10412ca00 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe1098d0640 1 --2- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe1040718e0 0x7fe1041c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:25.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.676+0000 7fe10a0d1640 1 --2- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe10412fb30 0x7fe1041c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:25.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.677+0000 7fe1090cf640 1 --2- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe10412ca00 0x7fe1041c63f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe10000e9f0 tx=0x7fe10000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:25.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.677+0000 7fe0f67fc640 1 -- 172.21.15.27:0/1838043585 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10000ccc0 con 0x7fe10412ca00 2024-09-21T03:28:25.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.677+0000 7fe0f67fc640 1 -- 172.21.15.27:0/1838043585 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe100010430 con 0x7fe10412ca00 2024-09-21T03:28:25.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.677+0000 7fe10bb34640 1 -- 172.21.15.27:0/1838043585 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe10409b480 con 0x7fe10412ca00 2024-09-21T03:28:25.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.677+0000 7fe0f67fc640 1 -- 172.21.15.27:0/1838043585 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe100010860 con 0x7fe10412ca00 2024-09-21T03:28:25.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.677+0000 7fe10bb34640 1 -- 172.21.15.27:0/1838043585 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe10409ba20 con 0x7fe10412ca00 2024-09-21T03:28:25.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.678+0000 7fe10bb34640 1 -- 172.21.15.27:0/1838043585 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe10412c4c0 con 0x7fe10412ca00 2024-09-21T03:28:25.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.678+0000 7fe0f67fc640 1 -- 172.21.15.27:0/1838043585 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7fe100010a00 con 0x7fe10412ca00 2024-09-21T03:28:25.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.679+0000 7fe1098d0640 1 -- 172.21.15.27:0/1838043585 >> v1:172.21.15.27:6800/1098903051 conn(0x7fe0e8047fd0 legacy=0x7fe0e804a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:25.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.679+0000 7fe0f67fc640 1 -- 172.21.15.27:0/1838043585 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7fe10005e390 con 0x7fe10412ca00 2024-09-21T03:28:25.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.683+0000 7fe0f67fc640 1 -- 172.21.15.27:0/1838043585 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+152684 (secure 0 0 0) 0x7fe10005d520 con 0x7fe10412ca00 2024-09-21T03:28:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.870+0000 7fe10bb34640 1 -- 172.21.15.27:0/1838043585 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fe104129d20 con 0x7fe10412ca00 2024-09-21T03:28:25.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.872+0000 7fe0f67fc640 1 -- 172.21.15.27:0/1838043585 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+108901 (secure 0 0 0) 0x7fe10003d6f0 con 0x7fe10412ca00 2024-09-21T03:28:25.876 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:25.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.875+0000 7fe10bb34640 1 -- 172.21.15.27:0/1838043585 >> v1:172.21.15.27:6800/1098903051 conn(0x7fe0e8047fd0 legacy=0x7fe0e804a4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:25.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.875+0000 7fe10bb34640 1 -- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe10412ca00 msgr2=0x7fe1041c63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:25.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.875+0000 7fe10bb34640 1 --2- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe10412ca00 0x7fe1041c63f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe10000e9f0 tx=0x7fe10000eeb0 comp rx=0 tx=0).stop 2024-09-21T03:28:25.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.875+0000 7fe10bb34640 1 -- 172.21.15.27:0/1838043585 shutdown_connections 2024-09-21T03:28:25.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.875+0000 7fe10bb34640 1 --2- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe10412fb30 0x7fe1041c6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:25.878 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.875+0000 7fe10bb34640 1 --2- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe10412ca00 0x7fe1041c63f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:25.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.875+0000 7fe10bb34640 1 --2- 172.21.15.27:0/1838043585 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe1040718e0 0x7fe1041c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:25.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.876+0000 7fe10bb34640 1 -- 172.21.15.27:0/1838043585 >> 172.21.15.27:0/1838043585 conn(0x7fe1040a1d70 msgr2=0x7fe104137960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:25.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.876+0000 7fe10bb34640 1 -- 172.21.15.27:0/1838043585 shutdown_connections 2024-09-21T03:28:25.879 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:25.876+0000 7fe10bb34640 1 -- 172.21.15.27:0/1838043585 wait complete. 2024-09-21T03:28:25.892 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6800","nonce":1098903051}]},"active_addr":"172.21.15.27:6800/1098903051","active_change":"2024-09-21T03:26:57.495177+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4099,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1012963840}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3140689679}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1436650201}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3459873364}]}]} 2024-09-21T03:28:25.895 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T03:28:25.895 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T03:28:25.895 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:28:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.011+0000 7f3c7556a640 1 -- 172.21.15.27:0/947962644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7012a220 msgr2=0x7f3c7012a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.011+0000 7f3c7556a640 1 --2- 172.21.15.27:0/947962644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7012a220 0x7f3c7012a630 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009860 tx=0x7f3c6002a410 comp rx=0 tx=0).stop 2024-09-21T03:28:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.011+0000 7f3c7556a640 1 -- 172.21.15.27:0/947962644 shutdown_connections 2024-09-21T03:28:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.011+0000 7f3c7556a640 1 --2- 172.21.15.27:0/947962644 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3c7012dbd0 0x7f3c7012dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.011+0000 7f3c7556a640 1 --2- 172.21.15.27:0/947962644 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3c7012ab70 0x7f3c7012d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.011+0000 7f3c7556a640 1 --2- 172.21.15.27:0/947962644 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7012a220 0x7f3c7012a630 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.011+0000 7f3c7556a640 1 -- 172.21.15.27:0/947962644 >> 172.21.15.27:0/947962644 conn(0x7f3c7009feb0 msgr2=0x7f3c700a02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.011+0000 7f3c7556a640 1 -- 172.21.15.27:0/947962644 shutdown_connections 2024-09-21T03:28:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.011+0000 7f3c7556a640 1 -- 172.21.15.27:0/947962644 wait complete. 2024-09-21T03:28:26.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.012+0000 7f3c7556a640 1 Processor -- start 2024-09-21T03:28:26.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.012+0000 7f3c7556a640 1 -- start start 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c7556a640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3c7012a220 0x7f3c70099870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c7556a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7012ab70 0x7f3c7006fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c7556a640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3c7012dbd0 0x7f3c70070120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c7556a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3c70135560 con 0x7f3c7012ab70 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c7556a640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3c701353e0 con 0x7f3c7012dbd0 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c7556a640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3c701356e0 con 0x7f3c7012a220 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c6e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7012ab70 0x7f3c7006fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c6f7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3c7012dbd0 0x7f3c70070120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c6effd640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3c7012a220 0x7f3c70099870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c6f7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3c7012dbd0 0x7f3c70070120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:44812/0 (socket says 172.21.15.27:44812) 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c6e7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7012ab70 0x7f3c7006fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56614/0 (socket says 172.21.15.27:56614) 2024-09-21T03:28:26.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.013+0000 7f3c6f7fe640 1 -- 172.21.15.27:0/1678601247 learned_addr learned my addr 172.21.15.27:0/1678601247 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c6effd640 1 -- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3c7012dbd0 msgr2=0x7f3c70070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c6effd640 1 --2- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3c7012dbd0 0x7f3c70070120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c6effd640 1 -- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7012ab70 msgr2=0x7f3c7006fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c6effd640 1 --2- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7012ab70 0x7f3c7006fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c6effd640 1 -- 172.21.15.27:0/1678601247 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c70127dc0 con 0x7f3c7012a220 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c6f7fe640 1 --2- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3c7012dbd0 0x7f3c70070120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c6e7fc640 1 --2- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7012ab70 0x7f3c7006fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c6effd640 1 --2- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3c7012a220 0x7f3c70099870 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3c7012e630 tx=0x7f3c60033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c577fe640 1 -- 172.21.15.27:0/1678601247 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60005c90 con 0x7f3c7012a220 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c577fe640 1 -- 172.21.15.27:0/1678601247 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c60033e10 con 0x7f3c7012a220 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c7556a640 1 -- 172.21.15.27:0/1678601247 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c70099db0 con 0x7f3c7012a220 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c577fe640 1 -- 172.21.15.27:0/1678601247 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60005260 con 0x7f3c7012a220 2024-09-21T03:28:26.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.014+0000 7f3c7556a640 1 -- 172.21.15.27:0/1678601247 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c70070840 con 0x7f3c7012a220 2024-09-21T03:28:26.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.015+0000 7f3c577fe640 1 -- 172.21.15.27:0/1678601247 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f3c60005400 con 0x7f3c7012a220 2024-09-21T03:28:26.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.016+0000 7f3c7556a640 1 -- 172.21.15.27:0/1678601247 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c70131e70 con 0x7f3c7012a220 2024-09-21T03:28:26.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.016+0000 7f3c6e7fc640 1 -- 172.21.15.27:0/1678601247 >> v1:172.21.15.27:6800/1098903051 conn(0x7f3c5c047fd0 legacy=0x7f3c5c04a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:26.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.016+0000 7f3c577fe640 1 -- 172.21.15.27:0/1678601247 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7f3c6007e850 con 0x7f3c7012a220 2024-09-21T03:28:26.022 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.020+0000 7f3c577fe640 1 -- 172.21.15.27:0/1678601247 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+152684 (secure 0 0 0) 0x7f3c6003f030 con 0x7f3c7012a220 2024-09-21T03:28:26.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.211+0000 7f3c7556a640 1 -- 172.21.15.27:0/1678601247 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3c7012d690 con 0x7f3c7012a220 2024-09-21T03:28:26.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.212+0000 7f3c577fe640 1 -- 172.21.15.27:0/1678601247 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+8820 (secure 0 0 0) 0x7f3c6003ad00 con 0x7f3c7012a220 2024-09-21T03:28:26.214 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:26.214 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:27:21.291221+0000","last_up_change":"2024-09-21T03:27:07.231513+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"squid","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T03:27:11.408521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6809","nonce":831015510}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6810","nonce":831015510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6812","nonce":831015510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6811","nonce":831015510}]},"public_addr":"172.21.15.27:6809/831015510","cluster_addr":"172.21.15.27:6810/831015510","heartbeat_back_addr":"172.21.15.27:6812/831015510","heartbeat_front_addr":"172.21.15.27:6811/831015510","state":["exists","up"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":1590434734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":1590434734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":1590434734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":1590434734}]},"public_addr":"172.21.15.27:6805/1590434734","cluster_addr":"172.21.15.27:6806/1590434734","heartbeat_back_addr":"172.21.15.27:6808/1590434734","heartbeat_front_addr":"172.21.15.27:6807/1590434734","state":["exists","up"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":3010908863}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":3010908863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":3010908863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":3010908863}]},"public_addr":"172.21.15.27:6801/3010908863","cluster_addr":"172.21.15.27:6802/3010908863","heartbeat_back_addr":"172.21.15.27:6804/3010908863","heartbeat_front_addr":"172.21.15.27:6803/3010908863","state":["exists","up"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1661572621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1661572621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1661572621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1661572621}]},"public_addr":"172.21.15.27:6813/1661572621","cluster_addr":"172.21.15.27:6814/1661572621","heartbeat_back_addr":"172.21.15.27:6816/1661572621","heartbeat_front_addr":"172.21.15.27:6815/1661572621","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:28:26.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.213+0000 7f3c7556a640 1 -- 172.21.15.27:0/1678601247 >> v1:172.21.15.27:6800/1098903051 conn(0x7f3c5c047fd0 legacy=0x7f3c5c04a4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:26.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.213+0000 7f3c7556a640 1 -- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3c7012a220 msgr2=0x7f3c70099870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.213+0000 7f3c7556a640 1 --2- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3c7012a220 0x7f3c70099870 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3c7012e630 tx=0x7f3c60033c20 comp rx=0 tx=0).stop 2024-09-21T03:28:26.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.214+0000 7f3c7556a640 1 -- 172.21.15.27:0/1678601247 shutdown_connections 2024-09-21T03:28:26.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.214+0000 7f3c7556a640 1 --2- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3c7012dbd0 0x7f3c70070120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.214+0000 7f3c7556a640 1 --2- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3c7012ab70 0x7f3c7006fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.214+0000 7f3c7556a640 1 --2- 172.21.15.27:0/1678601247 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3c7012a220 0x7f3c70099870 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.214+0000 7f3c7556a640 1 -- 172.21.15.27:0/1678601247 >> 172.21.15.27:0/1678601247 conn(0x7f3c7009feb0 msgr2=0x7f3c7009ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.214+0000 7f3c7556a640 1 -- 172.21.15.27:0/1678601247 shutdown_connections 2024-09-21T03:28:26.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.214+0000 7f3c7556a640 1 -- 172.21.15.27:0/1678601247 wait complete. 2024-09-21T03:28:26.228 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T03:28:26.228 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:28:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.342+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2795656487 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e240718e0 msgr2=0x7f5e2412c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.342+0000 7f5e2aa25640 1 --2- 172.21.15.27:0/2795656487 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e240718e0 0x7f5e2412c4c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009830 tx=0x7f5e1402a3f0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.342+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2795656487 shutdown_connections 2024-09-21T03:28:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.342+0000 7f5e2aa25640 1 --2- 172.21.15.27:0/2795656487 >> v2:172.21.15.27:3300/0 conn(0x7f5e2412fb30 0x7f5e2412ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.342+0000 7f5e2aa25640 1 --2- 172.21.15.27:0/2795656487 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e2412ca00 0x7f5e2412f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.342+0000 7f5e2aa25640 1 --2- 172.21.15.27:0/2795656487 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e240718e0 0x7f5e2412c4c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.343+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2795656487 >> 172.21.15.27:0/2795656487 conn(0x7f5e240a1d70 msgr2=0x7f5e240a2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.343+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2795656487 shutdown_connections 2024-09-21T03:28:26.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.343+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2795656487 wait complete. 2024-09-21T03:28:26.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.344+0000 7f5e2aa25640 1 Processor -- start 2024-09-21T03:28:26.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.345+0000 7f5e2aa25640 1 -- start start 2024-09-21T03:28:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e2aa25640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e240718e0 0x7f5e2409b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e2aa25640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e2412ca00 0x7f5e2409b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e2aa25640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e2412fb30 0x7f5e2409bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e2aa25640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5e241374a0 con 0x7f5e2412fb30 2024-09-21T03:28:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e2aa25640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f5e24137320 con 0x7f5e2412ca00 2024-09-21T03:28:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e2aa25640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f5e24137620 con 0x7f5e240718e0 2024-09-21T03:28:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e23fff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e240718e0 0x7f5e2409b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e23fff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e240718e0 0x7f5e2409b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60380/0 (socket says 172.21.15.27:60380) 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e23fff640 1 -- 172.21.15.27:0/2732006994 learned_addr learned my addr 172.21.15.27:0/2732006994 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e237fe640 1 --2- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e2412ca00 0x7f5e2409b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e28fc2640 1 --2- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e2412fb30 0x7f5e2409bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e23fff640 1 -- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e2412ca00 msgr2=0x7f5e2409b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e23fff640 1 --2- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e2412ca00 0x7f5e2409b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e23fff640 1 -- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e2412fb30 msgr2=0x7f5e2409bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e23fff640 1 --2- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e2412fb30 0x7f5e2409bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e23fff640 1 -- 172.21.15.27:0/2732006994 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e24133920 con 0x7f5e240718e0 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e28fc2640 1 --2- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e2412fb30 0x7f5e2409bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.346+0000 7f5e23fff640 1 --2- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e240718e0 0x7f5e2409b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009950 tx=0x7f5e14033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.347+0000 7f5e20ff9640 1 -- 172.21.15.27:0/2732006994 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14005c90 con 0x7f5e240718e0 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.347+0000 7f5e20ff9640 1 -- 172.21.15.27:0/2732006994 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e14031e50 con 0x7f5e240718e0 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.347+0000 7f5e20ff9640 1 -- 172.21.15.27:0/2732006994 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e14005260 con 0x7f5e240718e0 2024-09-21T03:28:26.350 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.347+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2732006994 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e24134180 con 0x7f5e240718e0 2024-09-21T03:28:26.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.347+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2732006994 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e24134f80 con 0x7f5e240718e0 2024-09-21T03:28:26.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.348+0000 7f5e20ff9640 1 -- 172.21.15.27:0/2732006994 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f5e14005400 con 0x7f5e240718e0 2024-09-21T03:28:26.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.348+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2732006994 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e2412f5f0 con 0x7f5e240718e0 2024-09-21T03:28:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.348+0000 7f5e237fe640 1 -- 172.21.15.27:0/2732006994 >> v1:172.21.15.27:6800/1098903051 conn(0x7f5e10047fd0 legacy=0x7f5e1004a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.349+0000 7f5e20ff9640 1 -- 172.21.15.27:0/2732006994 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7f5e1407e7c0 con 0x7f5e240718e0 2024-09-21T03:28:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.356+0000 7f5e20ff9640 1 -- 172.21.15.27:0/2732006994 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+152684 (secure 0 0 0) 0x7f5e1403f030 con 0x7f5e240718e0 2024-09-21T03:28:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.509+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2732006994 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e2406dda0 con 0x7f5e240718e0 2024-09-21T03:28:26.512 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.510+0000 7f5e20ff9640 1 -- 172.21.15.27:0/2732006994 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+8820 (secure 0 0 0) 0x7f5e14033e10 con 0x7f5e240718e0 2024-09-21T03:28:26.513 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:26.513 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:27:21.291221+0000","last_up_change":"2024-09-21T03:27:07.231513+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"squid","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T03:27:11.408521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6809","nonce":831015510}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6810","nonce":831015510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6812","nonce":831015510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6811","nonce":831015510}]},"public_addr":"172.21.15.27:6809/831015510","cluster_addr":"172.21.15.27:6810/831015510","heartbeat_back_addr":"172.21.15.27:6812/831015510","heartbeat_front_addr":"172.21.15.27:6811/831015510","state":["exists","up"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":1590434734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":1590434734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":1590434734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":1590434734}]},"public_addr":"172.21.15.27:6805/1590434734","cluster_addr":"172.21.15.27:6806/1590434734","heartbeat_back_addr":"172.21.15.27:6808/1590434734","heartbeat_front_addr":"172.21.15.27:6807/1590434734","state":["exists","up"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":3010908863}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":3010908863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":3010908863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":3010908863}]},"public_addr":"172.21.15.27:6801/3010908863","cluster_addr":"172.21.15.27:6802/3010908863","heartbeat_back_addr":"172.21.15.27:6804/3010908863","heartbeat_front_addr":"172.21.15.27:6803/3010908863","state":["exists","up"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1661572621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1661572621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1661572621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1661572621}]},"public_addr":"172.21.15.27:6813/1661572621","cluster_addr":"172.21.15.27:6814/1661572621","heartbeat_back_addr":"172.21.15.27:6816/1661572621","heartbeat_front_addr":"172.21.15.27:6815/1661572621","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:28:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.512+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2732006994 >> v1:172.21.15.27:6800/1098903051 conn(0x7f5e10047fd0 legacy=0x7f5e1004a4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.512+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e240718e0 msgr2=0x7f5e2409b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.512+0000 7f5e2aa25640 1 --2- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e240718e0 0x7f5e2409b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5e14009950 tx=0x7f5e14033c20 comp rx=0 tx=0).stop 2024-09-21T03:28:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.512+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2732006994 shutdown_connections 2024-09-21T03:28:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.512+0000 7f5e2aa25640 1 --2- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e2412fb30 0x7f5e2409bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.512+0000 7f5e2aa25640 1 --2- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e2412ca00 0x7f5e2409b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.512+0000 7f5e2aa25640 1 --2- 172.21.15.27:0/2732006994 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e240718e0 0x7f5e2409b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.512+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2732006994 >> 172.21.15.27:0/2732006994 conn(0x7f5e240a1d70 msgr2=0x7f5e24126c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.512+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2732006994 shutdown_connections 2024-09-21T03:28:26.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.512+0000 7f5e2aa25640 1 -- 172.21.15.27:0/2732006994 wait complete. 2024-09-21T03:28:26.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T03:28:26.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T03:28:26.527 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T03:28:26.528 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T03:28:26.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.668+0000 7f767a300640 1 -- 172.21.15.27:0/1960017459 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f767406ff70 msgr2=0x7f76741380b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.668+0000 7f767a300640 1 --2- 172.21.15.27:0/1960017459 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f767406ff70 0x7f76741380b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7668009900 tx=0x7f766802a440 comp rx=0 tx=0).stop 2024-09-21T03:28:26.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.668+0000 7f767a300640 1 -- 172.21.15.27:0/1960017459 shutdown_connections 2024-09-21T03:28:26.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.668+0000 7f767a300640 1 --2- 172.21.15.27:0/1960017459 >> v2:172.21.15.27:3300/0 conn(0x7f7674070380 0x7f767413a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.671 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.668+0000 7f767a300640 1 --2- 172.21.15.27:0/1960017459 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f767406ff70 0x7f76741380b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.668+0000 7f767a300640 1 --2- 172.21.15.27:0/1960017459 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f767406f620 0x7f767406fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.668+0000 7f767a300640 1 -- 172.21.15.27:0/1960017459 >> 172.21.15.27:0/1960017459 conn(0x7f767409d8f0 msgr2=0x7f767409dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.668+0000 7f767a300640 1 -- 172.21.15.27:0/1960017459 shutdown_connections 2024-09-21T03:28:26.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.668+0000 7f767a300640 1 -- 172.21.15.27:0/1960017459 wait complete. 2024-09-21T03:28:26.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.670+0000 7f767a300640 1 Processor -- start 2024-09-21T03:28:26.672 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.670+0000 7f767a300640 1 -- start start 2024-09-21T03:28:26.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.671+0000 7f767a300640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f767406f620 0x7f76740960f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.673 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.671+0000 7f767a300640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f767406ff70 0x7f7674096630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.671+0000 7f7673fff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f767406f620 0x7f76740960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.672+0000 7f7673fff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f767406f620 0x7f76740960f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60398/0 (socket says 172.21.15.27:60398) 2024-09-21T03:28:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.671+0000 7f76737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f767406ff70 0x7f7674096630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.672+0000 7f76737fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f767406ff70 0x7f7674096630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56636/0 (socket says 172.21.15.27:56636) 2024-09-21T03:28:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.672+0000 7f767a300640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7674070380 0x7f7674096b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.672+0000 7f7673fff640 1 -- 172.21.15.27:0/700827152 learned_addr learned my addr 172.21.15.27:0/700827152 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.672+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f767413b690 con 0x7f767406ff70 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.672+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f767413b510 con 0x7f7674070380 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.672+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f767413b810 con 0x7f767406f620 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.672+0000 7f767889d640 1 --2- 172.21.15.27:0/700827152 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7674070380 0x7f7674096b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.673+0000 7f76737fe640 1 -- 172.21.15.27:0/700827152 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f767406f620 msgr2=0x7f76740960f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.673+0000 7f76737fe640 1 --2- 172.21.15.27:0/700827152 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f767406f620 0x7f76740960f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.673+0000 7f76737fe640 1 -- 172.21.15.27:0/700827152 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7674070380 msgr2=0x7f7674096b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.673+0000 7f76737fe640 1 --2- 172.21.15.27:0/700827152 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7674070380 0x7f7674096b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.673+0000 7f76737fe640 1 -- 172.21.15.27:0/700827152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7674128c40 con 0x7f767406ff70 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.673+0000 7f767889d640 1 --2- 172.21.15.27:0/700827152 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7674070380 0x7f7674096b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.673+0000 7f7673fff640 1 --2- 172.21.15.27:0/700827152 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f767406f620 0x7f76740960f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.673+0000 7f76737fe640 1 --2- 172.21.15.27:0/700827152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f767406ff70 0x7f7674096630 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7668031000 tx=0x7f7668003c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.674+0000 7f7670ff9640 1 -- 172.21.15.27:0/700827152 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f766803f070 con 0x7f767406ff70 2024-09-21T03:28:26.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.674+0000 7f7670ff9640 1 -- 172.21.15.27:0/700827152 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7668004a90 con 0x7f767406ff70 2024-09-21T03:28:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.674+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7674127c20 con 0x7f767406ff70 2024-09-21T03:28:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.674+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7674125d30 con 0x7f767406ff70 2024-09-21T03:28:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.674+0000 7f7670ff9640 1 -- 172.21.15.27:0/700827152 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7668035030 con 0x7f767406ff70 2024-09-21T03:28:26.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.675+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7674076db0 con 0x7f767406ff70 2024-09-21T03:28:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.676+0000 7f7670ff9640 1 -- 172.21.15.27:0/700827152 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f7668042070 con 0x7f767406ff70 2024-09-21T03:28:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.676+0000 7f7670ff9640 1 -- 172.21.15.27:0/700827152 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7f766807f0e0 con 0x7f767406ff70 2024-09-21T03:28:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.676+0000 7f7670ff9640 1 -- 172.21.15.27:0/700827152 --> v1:172.21.15.27:6813/1661572621 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7668050e50 con 0x7f766004e6a0 2024-09-21T03:28:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.676+0000 7f7670ff9640 1 -- 172.21.15.27:0/700827152 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f766807f570 con 0x7f767406ff70 2024-09-21T03:28:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.676+0000 7f7673fff640 1 -- 172.21.15.27:0/700827152 >> v1:172.21.15.27:6800/1098903051 conn(0x7f7660047fd0 legacy=0x7f766004a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:26.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.677+0000 7f7670ff9640 1 -- 172.21.15.27:0/700827152 <== osd.3 v1:172.21.15.27:6813/1661572621 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7668050e50 con 0x7f766004e6a0 2024-09-21T03:28:26.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.695+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 --> v1:172.21.15.27:6813/1661572621 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7674126270 con 0x7f766004e6a0 2024-09-21T03:28:26.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.695+0000 7f7670ff9640 1 -- 172.21.15.27:0/700827152 <== osd.3 v1:172.21.15.27:6813/1661572621 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 4080920037) 0x7f7674126270 con 0x7f766004e6a0 2024-09-21T03:28:26.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.696+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 >> v1:172.21.15.27:6813/1661572621 conn(0x7f766004e6a0 legacy=0x7f7660050b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.696+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 >> v1:172.21.15.27:6800/1098903051 conn(0x7f7660047fd0 legacy=0x7f766004a4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.696+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f767406ff70 msgr2=0x7f7674096630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.696+0000 7f767a300640 1 --2- 172.21.15.27:0/700827152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f767406ff70 0x7f7674096630 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7668031000 tx=0x7f7668003c60 comp rx=0 tx=0).stop 2024-09-21T03:28:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.696+0000 7f767889d640 1 -- 172.21.15.27:0/700827152 reap_dead start 2024-09-21T03:28:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.696+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 shutdown_connections 2024-09-21T03:28:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.696+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 >> 172.21.15.27:0/700827152 conn(0x7f767409d8f0 msgr2=0x7f767412f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.696+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 shutdown_connections 2024-09-21T03:28:26.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.697+0000 7f767a300640 1 -- 172.21.15.27:0/700827152 wait complete. 2024-09-21T03:28:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.701+0000 7f1769b97640 1 -- 172.21.15.27:0/1026762529 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f17640712e0 msgr2=0x7f1764134cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.701+0000 7f1769b97640 1 --2- 172.21.15.27:0/1026762529 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f17640712e0 0x7f1764134cf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1758009860 tx=0x7f175802a3d0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.701+0000 7f1769b97640 1 -- 172.21.15.27:0/1026762529 shutdown_connections 2024-09-21T03:28:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.701+0000 7f1769b97640 1 --2- 172.21.15.27:0/1026762529 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1764135230 0x7f1764137650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.701+0000 7f1769b97640 1 --2- 172.21.15.27:0/1026762529 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f17640712e0 0x7f1764134cf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.701+0000 7f1769b97640 1 -- 172.21.15.27:0/1026762529 >> 172.21.15.27:0/1026762529 conn(0x7f176409eb10 msgr2=0x7f176409ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.701+0000 7f1769b97640 1 -- 172.21.15.27:0/1026762529 shutdown_connections 2024-09-21T03:28:26.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.702+0000 7f1769b97640 1 -- 172.21.15.27:0/1026762529 wait complete. 2024-09-21T03:28:26.705 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.703+0000 7f1769b97640 1 Processor -- start 2024-09-21T03:28:26.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.704+0000 7f1769b97640 1 -- start start 2024-09-21T03:28:26.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.705+0000 7f1769b97640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f17640707c0 0x7f1764096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.705+0000 7f1769b97640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f17640712e0 0x7f1764096fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.705+0000 7f1769b97640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1764135230 0x7f17640974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.705+0000 7f1769b97640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1764138750 con 0x7f1764135230 2024-09-21T03:28:26.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.705+0000 7f1769b97640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f17641385d0 con 0x7f17640707c0 2024-09-21T03:28:26.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.705+0000 7f1769b97640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f17641388d0 con 0x7f17640712e0 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.705+0000 7f1763fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1764135230 0x7f17640974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.705+0000 7f1763fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1764135230 0x7f17640974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56640/0 (socket says 172.21.15.27:56640) 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.705+0000 7f1763fff640 1 -- 172.21.15.27:0/2643925191 learned_addr learned my addr 172.21.15.27:0/2643925191 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.705+0000 7f1762ffd640 1 --2- 172.21.15.27:0/2643925191 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f17640712e0 0x7f1764096fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.705+0000 7f17637fe640 1 --2- 172.21.15.27:0/2643925191 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f17640707c0 0x7f1764096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.706+0000 7f1763fff640 1 -- 172.21.15.27:0/2643925191 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f17640712e0 msgr2=0x7f1764096fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.706+0000 7f1763fff640 1 --2- 172.21.15.27:0/2643925191 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f17640712e0 0x7f1764096fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.706+0000 7f1763fff640 1 -- 172.21.15.27:0/2643925191 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f17640707c0 msgr2=0x7f1764096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.706+0000 7f1763fff640 1 --2- 172.21.15.27:0/2643925191 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f17640707c0 0x7f1764096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.706+0000 7f1763fff640 1 -- 172.21.15.27:0/2643925191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1764095640 con 0x7f1764135230 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.706+0000 7f17637fe640 1 --2- 172.21.15.27:0/2643925191 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f17640707c0 0x7f1764096a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:28:26.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.706+0000 7f1763fff640 1 --2- 172.21.15.27:0/2643925191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1764135230 0x7f17640974e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f175c00ebf0 tx=0x7f175c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:26.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.706+0000 7f174bfff640 1 -- 172.21.15.27:0/2643925191 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c010630 con 0x7f1764135230 2024-09-21T03:28:26.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.707+0000 7f174bfff640 1 -- 172.21.15.27:0/2643925191 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f175c0107d0 con 0x7f1764135230 2024-09-21T03:28:26.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.707+0000 7f174bfff640 1 -- 172.21.15.27:0/2643925191 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c010be0 con 0x7f1764135230 2024-09-21T03:28:26.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.707+0000 7f1769b97640 1 -- 172.21.15.27:0/2643925191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1764096490 con 0x7f1764135230 2024-09-21T03:28:26.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.707+0000 7f1769b97640 1 -- 172.21.15.27:0/2643925191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f176412fab0 con 0x7f1764135230 2024-09-21T03:28:26.710 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.708+0000 7f1769b97640 1 -- 172.21.15.27:0/2643925191 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1764070bd0 con 0x7f1764135230 2024-09-21T03:28:26.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.709+0000 7f174bfff640 1 -- 172.21.15.27:0/2643925191 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f175c021430 con 0x7f1764135230 2024-09-21T03:28:26.712 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.710+0000 7f17637fe640 1 -- 172.21.15.27:0/2643925191 >> v1:172.21.15.27:6800/1098903051 conn(0x7f1750047fd0 legacy=0x7f175004a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:26.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.710+0000 7f174bfff640 1 -- 172.21.15.27:0/2643925191 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7f175c05e7e0 con 0x7f1764135230 2024-09-21T03:28:26.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.710+0000 7f174bfff640 1 -- 172.21.15.27:0/2643925191 --> v1:172.21.15.27:6801/3010908863 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f175c0020b0 con 0x7f175004e6a0 2024-09-21T03:28:26.713 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.710+0000 7f174bfff640 1 -- 172.21.15.27:0/2643925191 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f175c065050 con 0x7f1764135230 2024-09-21T03:28:26.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.712+0000 7f6920e07640 1 -- 172.21.15.27:0/948642991 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f691c096bd0 msgr2=0x7f691c096fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.712+0000 7f6920e07640 1 --2- 172.21.15.27:0/948642991 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f691c096bd0 0x7f691c096fe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6914005fd0 tx=0x7f691402b860 comp rx=0 tx=0).stop 2024-09-21T03:28:26.714 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.712+0000 7f174bfff640 1 -- 172.21.15.27:0/2643925191 <== osd.2 v1:172.21.15.27:6801/3010908863 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f175c0020b0 con 0x7f175004e6a0 2024-09-21T03:28:26.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.712+0000 7f6920e07640 1 -- 172.21.15.27:0/948642991 shutdown_connections 2024-09-21T03:28:26.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.712+0000 7f6920e07640 1 --2- 172.21.15.27:0/948642991 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f691c096bd0 0x7f691c096fe0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.712+0000 7f6920e07640 1 --2- 172.21.15.27:0/948642991 >> v2:172.21.15.27:3300/0 conn(0x7f691c12c100 0x7f691c12c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.712+0000 7f6920e07640 1 -- 172.21.15.27:0/948642991 >> 172.21.15.27:0/948642991 conn(0x7f691c09e5e0 msgr2=0x7f691c09aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.712+0000 7efe746e8640 1 -- 172.21.15.27:0/2450890140 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efe6c1301d0 msgr2=0x7efe6c1349f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.715 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.712+0000 7efe746e8640 1 --2- 172.21.15.27:0/2450890140 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efe6c1301d0 0x7efe6c1349f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efe6c12f9f0 tx=0x7efe5c02a400 comp rx=0 tx=0).stop 2024-09-21T03:28:26.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.713+0000 7f6920e07640 1 -- 172.21.15.27:0/948642991 shutdown_connections 2024-09-21T03:28:26.716 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.713+0000 7f6920e07640 1 -- 172.21.15.27:0/948642991 wait complete. 2024-09-21T03:28:26.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.713+0000 7efe746e8640 1 -- 172.21.15.27:0/2450890140 shutdown_connections 2024-09-21T03:28:26.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.713+0000 7efe746e8640 1 --2- 172.21.15.27:0/2450890140 >> v2:172.21.15.27:3300/0 conn(0x7efe6c134f30 0x7efe6c137350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.713+0000 7efe746e8640 1 --2- 172.21.15.27:0/2450890140 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efe6c1301d0 0x7efe6c1349f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.713+0000 7efe746e8640 1 -- 172.21.15.27:0/2450890140 >> 172.21.15.27:0/2450890140 conn(0x7efe6c0a1e10 msgr2=0x7efe6c0a2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.713+0000 7efe746e8640 1 -- 172.21.15.27:0/2450890140 shutdown_connections 2024-09-21T03:28:26.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.713+0000 7efe746e8640 1 -- 172.21.15.27:0/2450890140 wait complete. 2024-09-21T03:28:26.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.714+0000 7efe746e8640 1 Processor -- start 2024-09-21T03:28:26.717 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.714+0000 7f6920e07640 1 Processor -- start 2024-09-21T03:28:26.718 INFO:teuthology.orchestra.run.smithi027.stdout:47244640274 2024-09-21T03:28:26.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T03:28:26.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.714+0000 7f6920e07640 1 -- start start 2024-09-21T03:28:26.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.714+0000 7efe746e8640 1 -- start start 2024-09-21T03:28:26.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7efe746e8640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efe6c071720 0x7efe6c1c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.719 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7efe746e8640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efe6c1301d0 0x7efe6c1c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7f6920e07640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f691c096bd0 0x7f691c1c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7f6920e07640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f691c12c100 0x7f691c1c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7efe746e8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe6c134f30 0x7efe6c12b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7efe746e8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c138520 con 0x7efe6c134f30 2024-09-21T03:28:26.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7efe746e8640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7efe6c1383a0 con 0x7efe6c071720 2024-09-21T03:28:26.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7efe746e8640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7efe6c1386a0 con 0x7efe6c1301d0 2024-09-21T03:28:26.720 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7efe72484640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efe6c071720 0x7efe6c1c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7f6920e07640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f691c12ca50 0x7f691c1c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7f6920e07640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f691c134360 con 0x7f691c096bd0 2024-09-21T03:28:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7efe71c83640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efe6c1301d0 0x7efe6c1c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7efe72484640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efe6c071720 0x7efe6c1c6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:44858/0 (socket says 172.21.15.27:44858) 2024-09-21T03:28:26.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7efe72484640 1 -- 172.21.15.27:0/3457648052 learned_addr learned my addr 172.21.15.27:0/3457648052 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:26.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7f6920e07640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f691c06d420 con 0x7f691c12ca50 2024-09-21T03:28:26.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7f6920e07640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f691c1344e0 con 0x7f691c12c100 2024-09-21T03:28:26.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7f691a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f691c096bd0 0x7f691c1c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.722 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7f691a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f691c096bd0 0x7f691c1c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56664/0 (socket says 172.21.15.27:56664) 2024-09-21T03:28:26.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.715+0000 7f691a59c640 1 -- 172.21.15.27:0/754635130 learned_addr learned my addr 172.21.15.27:0/754635130 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:26.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7f6919d9b640 1 --2- 172.21.15.27:0/754635130 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f691c12c100 0x7f691c1c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7f691ad9d640 1 --2- 172.21.15.27:0/754635130 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f691c12ca50 0x7f691c1c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7efe71c83640 1 -- 172.21.15.27:0/3457648052 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efe6c071720 msgr2=0x7efe6c1c6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7efe71c83640 1 --2- 172.21.15.27:0/3457648052 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efe6c071720 0x7efe6c1c6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7efe71c83640 1 -- 172.21.15.27:0/3457648052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe6c134f30 msgr2=0x7efe6c12b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7f691a59c640 1 -- 172.21.15.27:0/754635130 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f691c12c100 msgr2=0x7f691c1c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7f691a59c640 1 --2- 172.21.15.27:0/754635130 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f691c12c100 0x7f691c1c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7f691a59c640 1 -- 172.21.15.27:0/754635130 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f691c12ca50 msgr2=0x7f691c1c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7f691a59c640 1 --2- 172.21.15.27:0/754635130 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f691c12ca50 0x7f691c1c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7f691a59c640 1 -- 172.21.15.27:0/754635130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f691c128bb0 con 0x7f691c096bd0 2024-09-21T03:28:26.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7f6919d9b640 1 --2- 172.21.15.27:0/754635130 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f691c12c100 0x7f691c1c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:28:26.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7efe71c83640 1 --2- 172.21.15.27:0/3457648052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe6c134f30 0x7efe6c12b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7efe71c83640 1 -- 172.21.15.27:0/3457648052 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6c099ba0 con 0x7efe6c1301d0 2024-09-21T03:28:26.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.717+0000 7f691a59c640 1 --2- 172.21.15.27:0/754635130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f691c096bd0 0x7f691c1c5770 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f691000e9f0 tx=0x7f691000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:26.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7efe72484640 1 --2- 172.21.15.27:0/3457648052 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efe6c071720 0x7efe6c1c6d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:26.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.716+0000 7efe71c83640 1 --2- 172.21.15.27:0/3457648052 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efe6c1301d0 0x7efe6c1c7250 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7efe5c031000 tx=0x7efe5c005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:26.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.717+0000 7efe62ffd640 1 -- 172.21.15.27:0/3457648052 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c037040 con 0x7efe6c1301d0 2024-09-21T03:28:26.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.717+0000 7efe62ffd640 1 -- 172.21.15.27:0/3457648052 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efe5c004130 con 0x7efe6c1301d0 2024-09-21T03:28:26.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.717+0000 7efe62ffd640 1 -- 172.21.15.27:0/3457648052 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5c0035f0 con 0x7efe6c1301d0 2024-09-21T03:28:26.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.717+0000 7f690affd640 1 -- 172.21.15.27:0/754635130 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691000cd80 con 0x7f691c096bd0 2024-09-21T03:28:26.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.718+0000 7efe746e8640 1 -- 172.21.15.27:0/3457648052 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe6c1c7790 con 0x7efe6c1301d0 2024-09-21T03:28:26.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.717+0000 7f690affd640 1 -- 172.21.15.27:0/754635130 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6910010430 con 0x7f691c096bd0 2024-09-21T03:28:26.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.717+0000 7f6920e07640 1 -- 172.21.15.27:0/754635130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f691c129c60 con 0x7f691c096bd0 2024-09-21T03:28:26.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.718+0000 7f6920e07640 1 -- 172.21.15.27:0/754635130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f691c12a880 con 0x7f691c096bd0 2024-09-21T03:28:26.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.718+0000 7f690affd640 1 -- 172.21.15.27:0/754635130 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910010840 con 0x7f691c096bd0 2024-09-21T03:28:26.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.718+0000 7f690affd640 1 -- 172.21.15.27:0/754635130 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f6910010a20 con 0x7f691c096bd0 2024-09-21T03:28:26.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.718+0000 7efe746e8640 1 -- 172.21.15.27:0/3457648052 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7efe6c12b7e0 con 0x7efe6c1301d0 2024-09-21T03:28:26.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.719+0000 7f6908ff9640 1 -- 172.21.15.27:0/754635130 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f691c096fe0 con 0x7f691c096bd0 2024-09-21T03:28:26.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.719+0000 7efe746e8640 1 -- 172.21.15.27:0/3457648052 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efe6c12fac0 con 0x7efe6c1301d0 2024-09-21T03:28:26.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.719+0000 7f6919d9b640 1 -- 172.21.15.27:0/754635130 >> v1:172.21.15.27:6800/1098903051 conn(0x7f68fc047fd0 legacy=0x7f68fc04a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:26.728 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.719+0000 7f690affd640 1 -- 172.21.15.27:0/754635130 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7f6910063cd0 con 0x7f691c096bd0 2024-09-21T03:28:26.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.719+0000 7f690affd640 1 -- 172.21.15.27:0/754635130 --> v1:172.21.15.27:6809/831015510 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f68fc0523a0 con 0x7f68fc04e6a0 2024-09-21T03:28:26.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.719+0000 7f690affd640 1 -- 172.21.15.27:0/754635130 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f6910064060 con 0x7f691c096bd0 2024-09-21T03:28:26.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.720+0000 7efe62ffd640 1 -- 172.21.15.27:0/3457648052 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7efe5c044020 con 0x7efe6c1301d0 2024-09-21T03:28:26.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.720+0000 7efe72484640 1 -- 172.21.15.27:0/3457648052 >> v1:172.21.15.27:6800/1098903051 conn(0x7efe50048020 legacy=0x7efe5004a530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:26.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.720+0000 7efe62ffd640 1 -- 172.21.15.27:0/3457648052 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7efe5c07e550 con 0x7efe6c1301d0 2024-09-21T03:28:26.729 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.720+0000 7efe62ffd640 1 -- 172.21.15.27:0/3457648052 --> v1:172.21.15.27:6805/1590434734 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efe500523f0 con 0x7efe5004e6f0 2024-09-21T03:28:26.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.720+0000 7efe62ffd640 1 -- 172.21.15.27:0/3457648052 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7efe5c07e900 con 0x7efe6c1301d0 2024-09-21T03:28:26.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.721+0000 7f690affd640 1 -- 172.21.15.27:0/754635130 <== osd.0 v1:172.21.15.27:6809/831015510 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f68fc0523a0 con 0x7f68fc04e6a0 2024-09-21T03:28:26.730 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.721+0000 7efe62ffd640 1 -- 172.21.15.27:0/3457648052 <== osd.1 v1:172.21.15.27:6805/1590434734 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7efe500523f0 con 0x7efe5004e6f0 2024-09-21T03:28:26.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.742+0000 7f1769b97640 1 -- 172.21.15.27:0/2643925191 --> v1:172.21.15.27:6801/3010908863 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1764135890 con 0x7f175004e6a0 2024-09-21T03:28:26.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.743+0000 7f174bfff640 1 -- 172.21.15.27:0/2643925191 <== osd.2 v1:172.21.15.27:6801/3010908863 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 4080920037) 0x7f1764135890 con 0x7f175004e6a0 2024-09-21T03:28:26.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.743+0000 7f1769b97640 1 -- 172.21.15.27:0/2643925191 >> v1:172.21.15.27:6801/3010908863 conn(0x7f175004e6a0 legacy=0x7f1750050b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.743+0000 7f1769b97640 1 -- 172.21.15.27:0/2643925191 >> v1:172.21.15.27:6800/1098903051 conn(0x7f1750047fd0 legacy=0x7f175004a4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:26.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.744+0000 7f1769b97640 1 -- 172.21.15.27:0/2643925191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1764135230 msgr2=0x7f17640974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.744+0000 7f1769b97640 1 --2- 172.21.15.27:0/2643925191 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1764135230 0x7f17640974e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f175c00ebf0 tx=0x7f175c00c6a0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.744+0000 7f1763fff640 1 -- 172.21.15.27:0/2643925191 reap_dead start 2024-09-21T03:28:26.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.744+0000 7f1769b97640 1 -- 172.21.15.27:0/2643925191 shutdown_connections 2024-09-21T03:28:26.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.744+0000 7f1769b97640 1 -- 172.21.15.27:0/2643925191 >> 172.21.15.27:0/2643925191 conn(0x7f176409eb10 msgr2=0x7f176409aad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.744+0000 7f1769b97640 1 -- 172.21.15.27:0/2643925191 shutdown_connections 2024-09-21T03:28:26.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.744+0000 7f1769b97640 1 -- 172.21.15.27:0/2643925191 wait complete. 2024-09-21T03:28:26.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.753+0000 7f6908ff9640 1 -- 172.21.15.27:0/754635130 --> v1:172.21.15.27:6809/831015510 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f691c12ad60 con 0x7f68fc04e6a0 2024-09-21T03:28:26.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.753+0000 7efe746e8640 1 -- 172.21.15.27:0/3457648052 --> v1:172.21.15.27:6805/1590434734 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efe6c12bc60 con 0x7efe5004e6f0 2024-09-21T03:28:26.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.754+0000 7f690affd640 1 -- 172.21.15.27:0/754635130 <== osd.0 v1:172.21.15.27:6809/831015510 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 22439654) 0x7f691c12ad60 con 0x7f68fc04e6a0 2024-09-21T03:28:26.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.754+0000 7f6908ff9640 1 -- 172.21.15.27:0/754635130 >> v1:172.21.15.27:6809/831015510 conn(0x7f68fc04e6a0 legacy=0x7f68fc050b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.754+0000 7efe62ffd640 1 -- 172.21.15.27:0/3457648052 <== osd.1 v1:172.21.15.27:6805/1590434734 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3077695196) 0x7efe6c12bc60 con 0x7efe5004e6f0 2024-09-21T03:28:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.754+0000 7f6908ff9640 1 -- 172.21.15.27:0/754635130 >> v1:172.21.15.27:6800/1098903051 conn(0x7f68fc047fd0 legacy=0x7f68fc04a4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.754+0000 7f6908ff9640 1 -- 172.21.15.27:0/754635130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f691c096bd0 msgr2=0x7f691c1c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.754+0000 7f6908ff9640 1 --2- 172.21.15.27:0/754635130 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f691c096bd0 0x7f691c1c5770 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f691000e9f0 tx=0x7f691000eeb0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.754+0000 7f691ad9d640 1 -- 172.21.15.27:0/754635130 reap_dead start 2024-09-21T03:28:26.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7f6908ff9640 1 -- 172.21.15.27:0/754635130 shutdown_connections 2024-09-21T03:28:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7efe746e8640 1 -- 172.21.15.27:0/3457648052 >> v1:172.21.15.27:6805/1590434734 conn(0x7efe5004e6f0 legacy=0x7efe50050b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7f6908ff9640 1 -- 172.21.15.27:0/754635130 >> 172.21.15.27:0/754635130 conn(0x7f691c09e5e0 msgr2=0x7f691c098af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7efe746e8640 1 -- 172.21.15.27:0/3457648052 >> v1:172.21.15.27:6800/1098903051 conn(0x7efe50048020 legacy=0x7efe5004a530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:26.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7efe746e8640 1 -- 172.21.15.27:0/3457648052 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efe6c1301d0 msgr2=0x7efe6c1c7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7efe746e8640 1 --2- 172.21.15.27:0/3457648052 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efe6c1301d0 0x7efe6c1c7250 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7efe5c031000 tx=0x7efe5c005c90 comp rx=0 tx=0).stop 2024-09-21T03:28:26.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7f6908ff9640 1 -- 172.21.15.27:0/754635130 shutdown_connections 2024-09-21T03:28:26.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7efe72c85640 1 -- 172.21.15.27:0/3457648052 reap_dead start 2024-09-21T03:28:26.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7f6908ff9640 1 -- 172.21.15.27:0/754635130 wait complete. 2024-09-21T03:28:26.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7efe746e8640 1 -- 172.21.15.27:0/3457648052 shutdown_connections 2024-09-21T03:28:26.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7efe746e8640 1 -- 172.21.15.27:0/3457648052 >> 172.21.15.27:0/3457648052 conn(0x7efe6c0a1e10 msgr2=0x7efe6c12edf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7efe746e8640 1 -- 172.21.15.27:0/3457648052 shutdown_connections 2024-09-21T03:28:26.760 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.755+0000 7efe746e8640 1 -- 172.21.15.27:0/3457648052 wait complete. 2024-09-21T03:28:26.764 INFO:teuthology.orchestra.run.smithi027.stdout:47244640274 2024-09-21T03:28:26.764 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T03:28:26.775 INFO:teuthology.orchestra.run.smithi027.stdout:38654705683 2024-09-21T03:28:26.776 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T03:28:26.776 INFO:teuthology.orchestra.run.smithi027.stdout:47244640275 2024-09-21T03:28:26.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T03:28:26.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.854+0000 7f6105e99640 1 -- 172.21.15.27:0/1901476185 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f610012c100 msgr2=0x7f610012c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.854+0000 7f6105e99640 1 --2- 172.21.15.27:0/1901476185 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f610012c100 0x7f610012c510 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009830 tx=0x7f60f002a400 comp rx=0 tx=0).stop 2024-09-21T03:28:26.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.855+0000 7f6105e99640 1 -- 172.21.15.27:0/1901476185 shutdown_connections 2024-09-21T03:28:26.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.855+0000 7f6105e99640 1 --2- 172.21.15.27:0/1901476185 >> v2:172.21.15.27:3300/0 conn(0x7f6100096bd0 0x7f6100096fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.855+0000 7f6105e99640 1 --2- 172.21.15.27:0/1901476185 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f610012c100 0x7f610012c510 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.855+0000 7f6105e99640 1 -- 172.21.15.27:0/1901476185 >> 172.21.15.27:0/1901476185 conn(0x7f610009e5e0 msgr2=0x7f610009aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.855+0000 7f6105e99640 1 -- 172.21.15.27:0/1901476185 shutdown_connections 2024-09-21T03:28:26.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.855+0000 7f6105e99640 1 -- 172.21.15.27:0/1901476185 wait complete. 2024-09-21T03:28:26.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.856+0000 7f6105e99640 1 Processor -- start 2024-09-21T03:28:26.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.857+0000 7f6105e99640 1 -- start start 2024-09-21T03:28:26.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.858+0000 7f6105e99640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6100096bd0 0x7f61001c3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.858+0000 7f60ff7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6100096bd0 0x7f61001c3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.858+0000 7f60ff7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6100096bd0 0x7f61001c3570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:44884/0 (socket says 172.21.15.27:44884) 2024-09-21T03:28:26.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.858+0000 7f60ff7fe640 1 -- 172.21.15.27:0/2317194505 learned_addr learned my addr 172.21.15.27:0/2317194505 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:26.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.858+0000 7f6105e99640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f610012c100 0x7f61001c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.858+0000 7f6105e99640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f610012ca50 0x7f61001c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.859+0000 7f60feffd640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f610012c100 0x7f61001c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.859+0000 7f6105e99640 1 -- 172.21.15.27:0/2317194505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6100134360 con 0x7f610012c100 2024-09-21T03:28:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.859+0000 7f6105e99640 1 -- 172.21.15.27:0/2317194505 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f610006d430 con 0x7f6100096bd0 2024-09-21T03:28:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.859+0000 7f6105e99640 1 -- 172.21.15.27:0/2317194505 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f61001344e0 con 0x7f610012ca50 2024-09-21T03:28:26.862 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.859+0000 7f60fffff640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f610012ca50 0x7f61001c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.860+0000 7f60feffd640 1 -- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f610012ca50 msgr2=0x7f61001c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.863 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.860+0000 7f60feffd640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f610012ca50 0x7f61001c3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.860+0000 7f60feffd640 1 -- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6100096bd0 msgr2=0x7f61001c3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.860+0000 7f60feffd640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6100096bd0 0x7f61001c3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.860+0000 7f60feffd640 1 -- 172.21.15.27:0/2317194505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6100123660 con 0x7f610012c100 2024-09-21T03:28:26.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.860+0000 7f60feffd640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f610012c100 0x7f61001c3ab0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f610006e000 tx=0x7f60f400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:26.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.861+0000 7f60ff7fe640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6100096bd0 0x7f61001c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:28:26.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.861+0000 7f60e7fff640 1 -- 172.21.15.27:0/2317194505 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f4005070 con 0x7f610012c100 2024-09-21T03:28:26.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.861+0000 7f60e7fff640 1 -- 172.21.15.27:0/2317194505 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f60f4005210 con 0x7f610012c100 2024-09-21T03:28:26.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.861+0000 7f6105e99640 1 -- 172.21.15.27:0/2317194505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6100123f00 con 0x7f610012c100 2024-09-21T03:28:26.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.862+0000 7f60e7fff640 1 -- 172.21.15.27:0/2317194505 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f400a870 con 0x7f610012c100 2024-09-21T03:28:26.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.862+0000 7f6105e99640 1 -- 172.21.15.27:0/2317194505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61000942b0 con 0x7f610012c100 2024-09-21T03:28:26.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.863+0000 7f60e7fff640 1 -- 172.21.15.27:0/2317194505 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f60f400aad0 con 0x7f610012c100 2024-09-21T03:28:26.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.863+0000 7f60ff7fe640 1 -- 172.21.15.27:0/2317194505 >> v1:172.21.15.27:6800/1098903051 conn(0x7f60ec047fd0 legacy=0x7f60ec04a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:26.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.863+0000 7f60e7fff640 1 -- 172.21.15.27:0/2317194505 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7f60f403bdc0 con 0x7f610012c100 2024-09-21T03:28:26.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.864+0000 7f60e5ffb640 1 -- 172.21.15.27:0/2317194505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d8005290 con 0x7f610012c100 2024-09-21T03:28:26.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.872+0000 7f60e7fff640 1 -- 172.21.15.27:0/2317194505 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+152684 (secure 0 0 0) 0x7f60f4005420 con 0x7f610012c100 2024-09-21T03:28:26.954 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.952+0000 7f7af527b640 1 -- 172.21.15.27:0/4103504650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae00047a0 con 0x7f7af012ca50 2024-09-21T03:28:26.955 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.953+0000 7f7af84e1640 1 -- 172.21.15.27:0/4103504650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af012ca50 msgr2=0x7f7af00964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.953+0000 7f7af84e1640 1 --2- 172.21.15.27:0/4103504650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af012ca50 0x7f7af00964c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0005fd0 tx=0x7f7ae002a420 comp rx=0 tx=0).stop 2024-09-21T03:28:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.953+0000 7f7af84e1640 1 -- 172.21.15.27:0/4103504650 shutdown_connections 2024-09-21T03:28:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.953+0000 7f7af84e1640 1 --2- 172.21.15.27:0/4103504650 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7af0096a00 0x7f7af0096e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.953+0000 7f7af84e1640 1 --2- 172.21.15.27:0/4103504650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af012ca50 0x7f7af00964c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.953+0000 7f7af84e1640 1 --2- 172.21.15.27:0/4103504650 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7af012c100 0x7f7af012c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.953+0000 7f7af84e1640 1 -- 172.21.15.27:0/4103504650 >> 172.21.15.27:0/4103504650 conn(0x7f7af009e5e0 msgr2=0x7f7af009aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.953+0000 7f7af84e1640 1 -- 172.21.15.27:0/4103504650 shutdown_connections 2024-09-21T03:28:26.956 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.953+0000 7f7af84e1640 1 -- 172.21.15.27:0/4103504650 wait complete. 2024-09-21T03:28:26.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.954+0000 7f7af84e1640 1 Processor -- start 2024-09-21T03:28:26.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.955+0000 7f7af84e1640 1 -- start start 2024-09-21T03:28:26.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.955+0000 7f7af84e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af0096a00 0x7f7af01c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.957 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.955+0000 7f7af84e1640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7af012c100 0x7f7af01c59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.955+0000 7f7af84e1640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7af012ca50 0x7f7af01c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:26.958 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.955+0000 7f7af84e1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7af006d3e0 con 0x7f7af0096a00 2024-09-21T03:28:26.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.955+0000 7f7af84e1640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f7af006d260 con 0x7f7af012ca50 2024-09-21T03:28:26.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af84e1640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f7af0133350 con 0x7f7af012c100 2024-09-21T03:28:26.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af5a7c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7af012c100 0x7f7af01c59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af5a7c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7af012c100 0x7f7af01c59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60506/0 (socket says 172.21.15.27:60506) 2024-09-21T03:28:26.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af5a7c640 1 -- 172.21.15.27:0/1376312309 learned_addr learned my addr 172.21.15.27:0/1376312309 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:26.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af6a7e640 1 --2- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7af012ca50 0x7f7af01c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af627d640 1 --2- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af0096a00 0x7f7af01c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:26.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af5a7c640 1 -- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7af012ca50 msgr2=0x7f7af01c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af5a7c640 1 --2- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7af012ca50 0x7f7af01c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.959 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af5a7c640 1 -- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af0096a00 msgr2=0x7f7af01c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:26.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af5a7c640 1 --2- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af0096a00 0x7f7af01c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:26.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af5a7c640 1 -- 172.21.15.27:0/1376312309 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7af0128450 con 0x7f7af012c100 2024-09-21T03:28:26.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.956+0000 7f7af627d640 1 --2- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af0096a00 0x7f7af01c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:28:26.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.957+0000 7f7af6a7e640 1 --2- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7af012ca50 0x7f7af01c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:26.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.957+0000 7f7af5a7c640 1 --2- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7af012c100 0x7f7af01c59d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0006010 tx=0x7f7ae0003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:26.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.957+0000 7f7ae6ffd640 1 -- 172.21.15.27:0/1376312309 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae003e030 con 0x7f7af012c100 2024-09-21T03:28:26.960 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.957+0000 7f7ae6ffd640 1 -- 172.21.15.27:0/1376312309 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ae00047a0 con 0x7f7af012c100 2024-09-21T03:28:26.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.957+0000 7f7ae6ffd640 1 -- 172.21.15.27:0/1376312309 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0035040 con 0x7f7af012c100 2024-09-21T03:28:26.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.957+0000 7f7af84e1640 1 -- 172.21.15.27:0/1376312309 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7af0127de0 con 0x7f7af012c100 2024-09-21T03:28:26.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.957+0000 7f7af84e1640 1 -- 172.21.15.27:0/1376312309 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f7af01c45f0 con 0x7f7af012c100 2024-09-21T03:28:26.961 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.959+0000 7f7ae4ff9640 1 -- 172.21.15.27:0/1376312309 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af006a2e0 con 0x7f7af012c100 2024-09-21T03:28:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.959+0000 7f7ae6ffd640 1 -- 172.21.15.27:0/1376312309 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f7ae0041030 con 0x7f7af012c100 2024-09-21T03:28:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.960+0000 7f7af627d640 1 -- 172.21.15.27:0/1376312309 >> v1:172.21.15.27:6800/1098903051 conn(0x7f7ad4048000 legacy=0x7f7ad404a510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:26.962 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.960+0000 7f7ae6ffd640 1 -- 172.21.15.27:0/1376312309 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7f7ae007ddf0 con 0x7f7af012c100 2024-09-21T03:28:26.969 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.967+0000 7f7ae6ffd640 1 -- 172.21.15.27:0/1376312309 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+152684 (secure 0 0 0) 0x7f7ae007cf80 con 0x7f7af012c100 2024-09-21T03:28:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.998+0000 7facf2234640 1 -- 172.21.15.27:0/1550325475 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facd4004800 msgr2=0x7facd4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.998+0000 7facf2234640 1 --2- 172.21.15.27:0/1550325475 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facd4004800 0x7facd4004c10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7facec070050 tx=0x7facd4005830 comp rx=0 tx=0).stop 2024-09-21T03:28:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.998+0000 7facebfff640 1 -- 172.21.15.27:0/1550325475 reap_dead start 2024-09-21T03:28:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7facf2234640 1 -- 172.21.15.27:0/1550325475 shutdown_connections 2024-09-21T03:28:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7facf2234640 1 -- 172.21.15.27:0/1550325475 >> 172.21.15.27:0/1550325475 conn(0x7facec09eb80 msgr2=0x7facec09ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7facf2234640 1 -- 172.21.15.27:0/1550325475 shutdown_connections 2024-09-21T03:28:27.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7facf2234640 1 -- 172.21.15.27:0/1550325475 wait complete. 2024-09-21T03:28:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7f6dcad1b640 1 -- 172.21.15.27:0/4280486227 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6dc412c820 msgr2=0x7f6dc4137a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7f6dcad1b640 1 --2- 172.21.15.27:0/4280486227 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6dc412c820 0x7f6dc4137a10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6dac009860 tx=0x7f6dac02a460 comp rx=0 tx=0).stop 2024-09-21T03:28:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7f6dcad1b640 1 -- 172.21.15.27:0/4280486227 shutdown_connections 2024-09-21T03:28:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7f6dcad1b640 1 --2- 172.21.15.27:0/4280486227 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6dc412c820 0x7f6dc4137a10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7f6dcad1b640 1 --2- 172.21.15.27:0/4280486227 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6dc4071720 0x7f6dc412fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7f6dcad1b640 1 -- 172.21.15.27:0/4280486227 >> 172.21.15.27:0/4280486227 conn(0x7f6dc40a1e10 msgr2=0x7f6dc40a2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7f6dcad1b640 1 -- 172.21.15.27:0/4280486227 shutdown_connections 2024-09-21T03:28:27.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:26.999+0000 7f6dcad1b640 1 -- 172.21.15.27:0/4280486227 wait complete. 2024-09-21T03:28:27.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.001+0000 7facf2234640 1 Processor -- start 2024-09-21T03:28:27.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.001+0000 7facf2234640 1 -- start start 2024-09-21T03:28:27.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.001+0000 7f6dcad1b640 1 Processor -- start 2024-09-21T03:28:27.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.001+0000 7f6dcad1b640 1 -- start start 2024-09-21T03:28:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.002+0000 7facf2234640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facd4004800 0x7facec1c1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.002+0000 7faceb7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facd4004800 0x7facec1c1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:27.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.002+0000 7faceb7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facd4004800 0x7facec1c1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:44942/0 (socket says 172.21.15.27:44942) 2024-09-21T03:28:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.002+0000 7f6dcad1b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6dc4071720 0x7f6dc41c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.002+0000 7faceb7fe640 1 -- 172.21.15.27:0/2719823092 learned_addr learned my addr 172.21.15.27:0/2719823092 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:27.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.002+0000 7facf2234640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facec1c1bc0 0x7facec1c1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:27.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.002+0000 7facf2234640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facec1be1f0 0x7facec1be600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:27.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.002+0000 7faceaffd640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facec1c1bc0 0x7facec1c1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:27.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7facebfff640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facec1be1f0 0x7facec1be600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:27.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7f6dc8ab7640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6dc4071720 0x7f6dc41c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:27.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7facec06f070 con 0x7facec1be1f0 2024-09-21T03:28:27.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7facec06eef0 con 0x7facd4004800 2024-09-21T03:28:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7facec06f1f0 con 0x7facec1c1bc0 2024-09-21T03:28:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7f6dc8ab7640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6dc4071720 0x7f6dc41c5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60548/0 (socket says 172.21.15.27:60548) 2024-09-21T03:28:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7f6dc8ab7640 1 -- 172.21.15.27:0/732995787 learned_addr learned my addr 172.21.15.27:0/732995787 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:27.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7f6dcad1b640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6dc412c820 0x7f6dc41c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7faceaffd640 1 -- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facd4004800 msgr2=0x7facec1c1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7faceaffd640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facd4004800 0x7facec1c1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7f6dcad1b640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc4130000 0x7f6dc41c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.004+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc41386f0 con 0x7f6dc4130000 2024-09-21T03:28:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.004+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc4138570 con 0x7f6dc412c820 2024-09-21T03:28:27.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.004+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f6dc4138870 con 0x7f6dc4071720 2024-09-21T03:28:27.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.003+0000 7faceaffd640 1 -- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facec1be1f0 msgr2=0x7facec1be600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:27.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.004+0000 7faceaffd640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facec1be1f0 0x7facec1be600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.004+0000 7faceaffd640 1 -- 172.21.15.27:0/2719823092 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facec091670 con 0x7facec1c1bc0 2024-09-21T03:28:27.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.004+0000 7facebfff640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facec1be1f0 0x7facec1be600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:27.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.004+0000 7faceaffd640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facec1c1bc0 0x7facec1c1fd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7face0032000 tx=0x7face0005270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.004+0000 7f6dc92b8640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc4130000 0x7f6dc41c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.004+0000 7f6dc92b8640 1 -- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6dc4071720 msgr2=0x7f6dc41c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.004+0000 7f6dc3fff640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6dc412c820 0x7f6dc41c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:27.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7facd3fff640 1 -- 172.21.15.27:0/2719823092 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0037040 con 0x7facec1c1bc0 2024-09-21T03:28:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7f6dc92b8640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6dc4071720 0x7f6dc41c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7f6dc92b8640 1 -- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6dc412c820 msgr2=0x7f6dc41c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7f6dc92b8640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6dc412c820 0x7f6dc41c60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7f6dc92b8640 1 -- 172.21.15.27:0/732995787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc4134720 con 0x7f6dc4130000 2024-09-21T03:28:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7f6dc8ab7640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6dc4071720 0x7f6dc41c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:28:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7f6dc3fff640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6dc412c820 0x7f6dc41c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:28:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7facd3fff640 1 -- 172.21.15.27:0/2719823092 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7face002de30 con 0x7facec1c1bc0 2024-09-21T03:28:27.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7f6dc92b8640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc4130000 0x7f6dc41c65f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6dac031000 tx=0x7f6dac032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7facd3fff640 1 -- 172.21.15.27:0/2719823092 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7face0009880 con 0x7facec1c1bc0 2024-09-21T03:28:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7f6dc17fa640 1 -- 172.21.15.27:0/732995787 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dac040070 con 0x7f6dc4130000 2024-09-21T03:28:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7f6dc17fa640 1 -- 172.21.15.27:0/732995787 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6dac005310 con 0x7f6dc4130000 2024-09-21T03:28:27.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7f6dc17fa640 1 -- 172.21.15.27:0/732995787 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dac005c90 con 0x7f6dc4130000 2024-09-21T03:28:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.005+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc4131d20 con 0x7f6dc4130000 2024-09-21T03:28:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.006+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc4132190 con 0x7f6dc4130000 2024-09-21T03:28:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.006+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facec126b00 con 0x7facec1c1bc0 2024-09-21T03:28:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.006+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7facec1beb40 con 0x7facec1c1bc0 2024-09-21T03:28:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.008+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facec099480 con 0x7facec1c1bc0 2024-09-21T03:28:27.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.007+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc406fe00 con 0x7f6dc4130000 2024-09-21T03:28:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.008+0000 7f6dc17fa640 1 -- 172.21.15.27:0/732995787 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7f6dac039040 con 0x7f6dc4130000 2024-09-21T03:28:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.008+0000 7f6dc17fa640 1 -- 172.21.15.27:0/732995787 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7f6dac07ecc0 con 0x7f6dc4130000 2024-09-21T03:28:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.009+0000 7facd3fff640 1 -- 172.21.15.27:0/2719823092 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 5) ==== 55999+0+0 (secure 0 0 0) 0x7face0043020 con 0x7facec1c1bc0 2024-09-21T03:28:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.009+0000 7f6dc8ab7640 1 -- 172.21.15.27:0/732995787 >> v1:172.21.15.27:6800/1098903051 conn(0x7f6da4047fd0 legacy=0x7f6da404a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:27.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.009+0000 7facd3fff640 1 -- 172.21.15.27:0/2719823092 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(20..20 src has 1..20) ==== 2812+0+0 (secure 0 0 0) 0x7face007de90 con 0x7facec1c1bc0 2024-09-21T03:28:27.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.009+0000 7faceb7fe640 1 -- 172.21.15.27:0/2719823092 >> v1:172.21.15.27:6800/1098903051 conn(0x7facd8048030 legacy=0x7facd804a540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.016+0000 7f6dc17fa640 1 -- 172.21.15.27:0/732995787 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+152684 (secure 0 0 0) 0x7f6dc406fe00 con 0x7f6dc4130000 2024-09-21T03:28:27.018 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.016+0000 7facd3fff640 1 -- 172.21.15.27:0/2719823092 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+152684 (secure 0 0 0) 0x7facec099480 con 0x7facec1c1bc0 2024-09-21T03:28:27.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.064+0000 7f60ff7fe640 1 -- 172.21.15.27:0/2317194505 >> v1:172.21.15.27:6800/1098903051 conn(0x7f60ec047fd0 legacy=0x7f60ec04a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:27.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.134+0000 7f7ae4ff9640 1 -- 172.21.15.27:0/1376312309 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7af01c4f80 con 0x7f7af012c100 2024-09-21T03:28:27.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.134+0000 7f7ae6ffd640 1 -- 172.21.15.27:0/1376312309 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7ae0005d50 con 0x7f7af012c100 2024-09-21T03:28:27.137 INFO:teuthology.orchestra.run.smithi027.stdout:47244640274 2024-09-21T03:28:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.136+0000 7f7ae4ff9640 1 -- 172.21.15.27:0/1376312309 >> v1:172.21.15.27:6800/1098903051 conn(0x7f7ad4048000 legacy=0x7f7ad404a510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.136+0000 7f7ae4ff9640 1 -- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7af012c100 msgr2=0x7f7af01c59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.136+0000 7f7ae4ff9640 1 --2- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7af012c100 0x7f7af01c59d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0006010 tx=0x7f7ae0003ba0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.137+0000 7f7ae4ff9640 1 -- 172.21.15.27:0/1376312309 shutdown_connections 2024-09-21T03:28:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.137+0000 7f7ae4ff9640 1 --2- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7af012ca50 0x7f7af01c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.137+0000 7f7ae4ff9640 1 --2- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7af012c100 0x7f7af01c59d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.137+0000 7f7ae4ff9640 1 --2- 172.21.15.27:0/1376312309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7af0096a00 0x7f7af01c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.137+0000 7f7ae4ff9640 1 -- 172.21.15.27:0/1376312309 >> 172.21.15.27:0/1376312309 conn(0x7f7af009e5e0 msgr2=0x7f7af01350c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.137+0000 7f7ae4ff9640 1 -- 172.21.15.27:0/1376312309 shutdown_connections 2024-09-21T03:28:27.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.137+0000 7f7ae4ff9640 1 -- 172.21.15.27:0/1376312309 wait complete. 2024-09-21T03:28:27.155 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640275 got 47244640274 for osd.0 2024-09-21T03:28:27.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.175+0000 7f60e5ffb640 1 -- 172.21.15.27:0/2317194505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f60d8005850 con 0x7f610012c100 2024-09-21T03:28:27.178 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.176+0000 7f60e7fff640 1 -- 172.21.15.27:0/2317194505 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f60f400adc0 con 0x7f610012c100 2024-09-21T03:28:27.178 INFO:teuthology.orchestra.run.smithi027.stdout:47244640273 2024-09-21T03:28:27.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.178+0000 7f60e5ffb640 1 -- 172.21.15.27:0/2317194505 >> v1:172.21.15.27:6800/1098903051 conn(0x7f60ec047fd0 legacy=0x7f60ec04a4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:27.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.178+0000 7f60e5ffb640 1 -- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f610012c100 msgr2=0x7f61001c3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:27.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.178+0000 7f60e5ffb640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f610012c100 0x7f61001c3ab0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f610006e000 tx=0x7f60f400cb90 comp rx=0 tx=0).stop 2024-09-21T03:28:27.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.178+0000 7f60e5ffb640 1 -- 172.21.15.27:0/2317194505 shutdown_connections 2024-09-21T03:28:27.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.178+0000 7f60e5ffb640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f610012ca50 0x7f61001c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.178+0000 7f60e5ffb640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f610012c100 0x7f61001c3ab0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.178+0000 7f60e5ffb640 1 --2- 172.21.15.27:0/2317194505 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6100096bd0 0x7f61001c3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.178+0000 7f60e5ffb640 1 -- 172.21.15.27:0/2317194505 >> 172.21.15.27:0/2317194505 conn(0x7f610009e5e0 msgr2=0x7f61000720a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:27.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.179+0000 7f60e5ffb640 1 -- 172.21.15.27:0/2317194505 shutdown_connections 2024-09-21T03:28:27.183 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.179+0000 7f60e5ffb640 1 -- 172.21.15.27:0/2317194505 wait complete. 2024-09-21T03:28:27.198 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640274 got 47244640273 for osd.3 2024-09-21T03:28:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.210+0000 7faceb7fe640 1 -- 172.21.15.27:0/2719823092 >> v1:172.21.15.27:6800/1098903051 conn(0x7facd8048030 legacy=0x7facd804a540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:27.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.210+0000 7f6dc8ab7640 1 -- 172.21.15.27:0/732995787 >> v1:172.21.15.27:6800/1098903051 conn(0x7f6da4047fd0 legacy=0x7f6da404a4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.27:6800/1098903051 2024-09-21T03:28:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.262+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6dc4134d50 con 0x7f6dc4130000 2024-09-21T03:28:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.262+0000 7f6dc17fa640 1 -- 172.21.15.27:0/732995787 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6dac07dd30 con 0x7f6dc4130000 2024-09-21T03:28:27.265 INFO:teuthology.orchestra.run.smithi027.stdout:38654705681 2024-09-21T03:28:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.264+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 >> v1:172.21.15.27:6800/1098903051 conn(0x7f6da4047fd0 legacy=0x7f6da404a4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:27.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.264+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc4130000 msgr2=0x7f6dc41c65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:27.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.264+0000 7f6dcad1b640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc4130000 0x7f6dc41c65f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6dac031000 tx=0x7f6dac032c10 comp rx=0 tx=0).stop 2024-09-21T03:28:27.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.264+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 shutdown_connections 2024-09-21T03:28:27.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.264+0000 7f6dcad1b640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6dc4130000 0x7f6dc41c65f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.264+0000 7f6dcad1b640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6dc412c820 0x7f6dc41c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.264+0000 7f6dcad1b640 1 --2- 172.21.15.27:0/732995787 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6dc4071720 0x7f6dc41c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.264+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 >> 172.21.15.27:0/732995787 conn(0x7f6dc40a1e10 msgr2=0x7f6dc409c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:27.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.264+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 shutdown_connections 2024-09-21T03:28:27.267 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.264+0000 7f6dcad1b640 1 -- 172.21.15.27:0/732995787 wait complete. 2024-09-21T03:28:27.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705683 got 38654705681 for osd.1 2024-09-21T03:28:27.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.323+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7facec12a110 con 0x7facec1c1bc0 2024-09-21T03:28:27.326 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.324+0000 7facd3fff640 1 -- 172.21.15.27:0/2719823092 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7facec12a110 con 0x7facec1c1bc0 2024-09-21T03:28:27.326 INFO:teuthology.orchestra.run.smithi027.stdout:47244640273 2024-09-21T03:28:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.326+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 >> v1:172.21.15.27:6800/1098903051 conn(0x7facd8048030 legacy=0x7facd804a540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.326+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facec1c1bc0 msgr2=0x7facec1c1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:27.328 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.326+0000 7facf2234640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facec1c1bc0 0x7facec1c1fd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7face0032000 tx=0x7face0005270 comp rx=0 tx=0).stop 2024-09-21T03:28:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.326+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 shutdown_connections 2024-09-21T03:28:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.326+0000 7facf2234640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facec1be1f0 0x7facec1be600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.326+0000 7facf2234640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7facec1c1bc0 0x7facec1c1fd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.326+0000 7facf2234640 1 --2- 172.21.15.27:0/2719823092 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7facd4004800 0x7facec1c1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.326+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 >> 172.21.15.27:0/2719823092 conn(0x7facec09eb80 msgr2=0x7facec070850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.326+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 shutdown_connections 2024-09-21T03:28:27.329 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:27.326+0000 7facf2234640 1 -- 172.21.15.27:0/2719823092 wait complete. 2024-09-21T03:28:27.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640274 got 47244640273 for osd.2 2024-09-21T03:28:28.156 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T03:28:28.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T03:28:28.280 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T03:28:28.292 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.290+0000 7fc169dcb640 1 -- 172.21.15.27:0/3957260469 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc164072ba0 msgr2=0x7fc164072fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.290+0000 7fc169dcb640 1 --2- 172.21.15.27:0/3957260469 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc164072ba0 0x7fc164072fb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc15c00acb0 tx=0x7fc15c00a5d0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.290+0000 7fc169dcb640 1 -- 172.21.15.27:0/3957260469 shutdown_connections 2024-09-21T03:28:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.290+0000 7fc169dcb640 1 --2- 172.21.15.27:0/3957260469 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc1640734f0 0x7fc16412b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.290+0000 7fc169dcb640 1 --2- 172.21.15.27:0/3957260469 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc164072ba0 0x7fc164072fb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.290+0000 7fc169dcb640 1 -- 172.21.15.27:0/3957260469 >> 172.21.15.27:0/3957260469 conn(0x7fc164098360 msgr2=0x7fc164098770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.291+0000 7fc169dcb640 1 -- 172.21.15.27:0/3957260469 shutdown_connections 2024-09-21T03:28:28.293 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.291+0000 7fc169dcb640 1 -- 172.21.15.27:0/3957260469 wait complete. 2024-09-21T03:28:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.293+0000 7fc169dcb640 1 Processor -- start 2024-09-21T03:28:28.295 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.293+0000 7fc169dcb640 1 -- start start 2024-09-21T03:28:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.294+0000 7fc169dcb640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc164072ba0 0x7fc1641c1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.294+0000 7fc169dcb640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc1640734f0 0x7fc1641c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.294+0000 7fc169dcb640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc16412bda0 0x7fc1641c2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.294+0000 7fc169dcb640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc164134350 con 0x7fc16412bda0 2024-09-21T03:28:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.294+0000 7fc169dcb640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fc1641341d0 con 0x7fc164072ba0 2024-09-21T03:28:28.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.294+0000 7fc169dcb640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fc1641344d0 con 0x7fc1640734f0 2024-09-21T03:28:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.294+0000 7fc163fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc16412bda0 0x7fc1641c2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.294+0000 7fc163fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc16412bda0 0x7fc1641c2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56750/0 (socket says 172.21.15.27:56750) 2024-09-21T03:28:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.294+0000 7fc163fff640 1 -- 172.21.15.27:0/3577690611 learned_addr learned my addr 172.21.15.27:0/3577690611 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.294+0000 7fc162ffd640 1 --2- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc1640734f0 0x7fc1641c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.294+0000 7fc1637fe640 1 --2- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc164072ba0 0x7fc1641c1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.295+0000 7fc163fff640 1 -- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc1640734f0 msgr2=0x7fc1641c1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.295+0000 7fc163fff640 1 --2- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc1640734f0 0x7fc1641c1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.295+0000 7fc163fff640 1 -- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc164072ba0 msgr2=0x7fc1641c1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.295+0000 7fc163fff640 1 --2- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc164072ba0 0x7fc1641c1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.295+0000 7fc163fff640 1 -- 172.21.15.27:0/3577690611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc164090ad0 con 0x7fc16412bda0 2024-09-21T03:28:28.298 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.295+0000 7fc1637fe640 1 --2- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc164072ba0 0x7fc1641c1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.295+0000 7fc163fff640 1 --2- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc16412bda0 0x7fc1641c2100 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc15400c3c0 tx=0x7fc15400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.296+0000 7fc14bfff640 1 -- 172.21.15.27:0/3577690611 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154017070 con 0x7fc16412bda0 2024-09-21T03:28:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.296+0000 7fc169dcb640 1 -- 172.21.15.27:0/3577690611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc16408e0f0 con 0x7fc16412bda0 2024-09-21T03:28:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.296+0000 7fc169dcb640 1 -- 172.21.15.27:0/3577690611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc16408e640 con 0x7fc16412bda0 2024-09-21T03:28:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.296+0000 7fc14bfff640 1 -- 172.21.15.27:0/3577690611 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc1540053c0 con 0x7fc16412bda0 2024-09-21T03:28:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.296+0000 7fc14bfff640 1 -- 172.21.15.27:0/3577690611 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154003830 con 0x7fc16412bda0 2024-09-21T03:28:28.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.298+0000 7fc169dcb640 1 -- 172.21.15.27:0/3577690611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc164072fb0 con 0x7fc16412bda0 2024-09-21T03:28:28.301 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.299+0000 7fc14bfff640 1 -- 172.21.15.27:0/3577690611 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 6) ==== 59491+0+0 (secure 0 0 0) 0x7fc15401a070 con 0x7fc16412bda0 2024-09-21T03:28:28.302 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.300+0000 7fc14bfff640 1 -- 172.21.15.27:0/3577690611 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7fc154062670 con 0x7fc16412bda0 2024-09-21T03:28:28.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.306+0000 7fc14bfff640 1 -- 172.21.15.27:0/3577690611 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+152684 (secure 0 0 0) 0x7fc1540611c0 con 0x7fc16412bda0 2024-09-21T03:28:28.340 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T03:28:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.370+0000 7ff10f5d2640 1 -- 172.21.15.27:0/786685375 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff108131880 msgr2=0x7ff108133ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.370+0000 7ff10f5d2640 1 --2- 172.21.15.27:0/786685375 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff108131880 0x7ff108133ca0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff10000adb0 tx=0x7ff10002b740 comp rx=0 tx=0).stop 2024-09-21T03:28:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.371+0000 7ff10f5d2640 1 -- 172.21.15.27:0/786685375 shutdown_connections 2024-09-21T03:28:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.371+0000 7ff10f5d2640 1 --2- 172.21.15.27:0/786685375 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff108131880 0x7ff108133ca0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.371+0000 7ff10f5d2640 1 --2- 172.21.15.27:0/786685375 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff10812cc20 0x7ff108131340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.373 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.371+0000 7ff10f5d2640 1 -- 172.21.15.27:0/786685375 >> 172.21.15.27:0/786685375 conn(0x7ff10809e5e0 msgr2=0x7ff10809aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.372+0000 7ff10f5d2640 1 -- 172.21.15.27:0/786685375 shutdown_connections 2024-09-21T03:28:28.374 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.372+0000 7ff10f5d2640 1 -- 172.21.15.27:0/786685375 wait complete. 2024-09-21T03:28:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.373+0000 7ff10f5d2640 1 Processor -- start 2024-09-21T03:28:28.375 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.373+0000 7ff10f5d2640 1 -- start start 2024-09-21T03:28:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.374+0000 7ff10f5d2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff10812c100 0x7ff1081c1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.374+0000 7ff10f5d2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff10812cc20 0x7ff1081c1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.374+0000 7ff10d36e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff10812c100 0x7ff1081c1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.374+0000 7ff10cb6d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff10812cc20 0x7ff1081c1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.374+0000 7ff10cb6d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff10812cc20 0x7ff1081c1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:45052/0 (socket says 172.21.15.27:45052) 2024-09-21T03:28:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.374+0000 7ff10d36e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff10812c100 0x7ff1081c1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56816/0 (socket says 172.21.15.27:56816) 2024-09-21T03:28:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.374+0000 7ff10d36e640 1 -- 172.21.15.27:0/3194146176 learned_addr learned my addr 172.21.15.27:0/3194146176 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.374+0000 7ff10f5d2640 1 --2- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff108131880 0x7ff1081c2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.374+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff1081343f0 con 0x7ff10812c100 2024-09-21T03:28:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.375+0000 7ff10db6f640 1 --2- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff108131880 0x7ff1081c2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.377 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.375+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7ff10806d430 con 0x7ff10812cc20 2024-09-21T03:28:28.378 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.375+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7ff108134570 con 0x7ff108131880 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.375+0000 7ff10d36e640 1 -- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff108131880 msgr2=0x7ff1081c2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.375+0000 7ff10d36e640 1 --2- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff108131880 0x7ff1081c2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.375+0000 7ff10d36e640 1 -- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff10812cc20 msgr2=0x7ff1081c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.375+0000 7ff10d36e640 1 --2- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff10812cc20 0x7ff1081c1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.375+0000 7ff10d36e640 1 -- 172.21.15.27:0/3194146176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff108125d50 con 0x7ff10812c100 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.375+0000 7ff10cb6d640 1 --2- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff10812cc20 0x7ff1081c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.375+0000 7ff10db6f640 1 --2- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff108131880 0x7ff1081c2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.375+0000 7ff10d36e640 1 --2- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff10812c100 0x7ff1081c1810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff10400c7d0 tx=0x7ff10400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.376+0000 7ff0fdffb640 1 -- 172.21.15.27:0/3194146176 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff104015070 con 0x7ff10812c100 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.376+0000 7ff0fdffb640 1 -- 172.21.15.27:0/3194146176 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff104005360 con 0x7ff10812c100 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.376+0000 7ff0fdffb640 1 -- 172.21.15.27:0/3194146176 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff10400a8a0 con 0x7ff10812c100 2024-09-21T03:28:28.379 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.376+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1081254e0 con 0x7ff10812c100 2024-09-21T03:28:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.376+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff108123690 con 0x7ff10812c100 2024-09-21T03:28:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.378+0000 7ff0fdffb640 1 -- 172.21.15.27:0/3194146176 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 6) ==== 59491+0+0 (secure 0 0 0) 0x7ff104018070 con 0x7ff10812c100 2024-09-21T03:28:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.378+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff108124690 con 0x7ff10812c100 2024-09-21T03:28:28.380 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.378+0000 7ff0fdffb640 1 -- 172.21.15.27:0/3194146176 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7ff104062dd0 con 0x7ff10812c100 2024-09-21T03:28:28.384 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.382+0000 7ff0fdffb640 1 -- 172.21.15.27:0/3194146176 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+152684 (secure 0 0 0) 0x7ff104061cb0 con 0x7ff10812c100 2024-09-21T03:28:28.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.464+0000 7f7a92e48640 1 -- 172.21.15.27:0/2141490311 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7a8c132ea0 msgr2=0x7f7a8c1352c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.464+0000 7f7a92e48640 1 --2- 172.21.15.27:0/2141490311 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7a8c132ea0 0x7f7a8c1352c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7a84009be0 tx=0x7f7a8402a800 comp rx=0 tx=0).stop 2024-09-21T03:28:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.464+0000 7f7a92e48640 1 -- 172.21.15.27:0/2141490311 shutdown_connections 2024-09-21T03:28:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.464+0000 7f7a92e48640 1 --2- 172.21.15.27:0/2141490311 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7a8c132ea0 0x7f7a8c1352c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.464+0000 7f7a92e48640 1 --2- 172.21.15.27:0/2141490311 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7a8c12aaa0 0x7f7a8c132960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.464+0000 7f7a92e48640 1 -- 172.21.15.27:0/2141490311 >> 172.21.15.27:0/2141490311 conn(0x7f7a8c09fa20 msgr2=0x7f7a8c09fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.464+0000 7f7a92e48640 1 -- 172.21.15.27:0/2141490311 shutdown_connections 2024-09-21T03:28:28.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.464+0000 7f7a92e48640 1 -- 172.21.15.27:0/2141490311 wait complete. 2024-09-21T03:28:28.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.466+0000 7f7a92e48640 1 Processor -- start 2024-09-21T03:28:28.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.466+0000 7f7a92e48640 1 -- start start 2024-09-21T03:28:28.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.466+0000 7f7a92e48640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7a8c073c10 0x7f7a8c12e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.466+0000 7f7a92e48640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8c12aaa0 0x7f7a8c12ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a8bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8c12aaa0 0x7f7a8c12ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a8bfff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8c12aaa0 0x7f7a8c12ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56820/0 (socket says 172.21.15.27:56820) 2024-09-21T03:28:28.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a8bfff640 1 -- 172.21.15.27:0/2333492411 learned_addr learned my addr 172.21.15.27:0/2333492411 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:28.469 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a92e48640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7a8c132ea0 0x7f7a8c12f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a90be4640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7a8c073c10 0x7f7a8c12e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8c1365e0 con 0x7f7a8c12aaa0 2024-09-21T03:28:28.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f7a8c136460 con 0x7f7a8c073c10 2024-09-21T03:28:28.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f7a8c136760 con 0x7f7a8c132ea0 2024-09-21T03:28:28.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a913e5640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7a8c132ea0 0x7f7a8c12f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a913e5640 1 -- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7a8c073c10 msgr2=0x7f7a8c12e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a913e5640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7a8c073c10 0x7f7a8c12e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a913e5640 1 -- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8c12aaa0 msgr2=0x7f7a8c12ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a913e5640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8c12aaa0 0x7f7a8c12ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a913e5640 1 -- 172.21.15.27:0/2333492411 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8c1316d0 con 0x7f7a8c132ea0 2024-09-21T03:28:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.467+0000 7f7a8bfff640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8c12aaa0 0x7f7a8c12ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.468+0000 7f7a90be4640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7a8c073c10 0x7f7a8c12e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:28:28.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.468+0000 7f7a913e5640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7a8c132ea0 0x7f7a8c12f1a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7a84009000 tx=0x7f7a84033dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.468+0000 7f7a897fa640 1 -- 172.21.15.27:0/2333492411 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a84042020 con 0x7f7a8c132ea0 2024-09-21T03:28:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.468+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a8c130fa0 con 0x7f7a8c132ea0 2024-09-21T03:28:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.468+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a8c099270 con 0x7f7a8c132ea0 2024-09-21T03:28:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.468+0000 7f7a897fa640 1 -- 172.21.15.27:0/2333492411 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7a84038040 con 0x7f7a8c132ea0 2024-09-21T03:28:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.468+0000 7f7a897fa640 1 -- 172.21.15.27:0/2333492411 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a84037070 con 0x7f7a8c132ea0 2024-09-21T03:28:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.470+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a6c005290 con 0x7f7a8c132ea0 2024-09-21T03:28:28.472 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.470+0000 7f7a897fa640 1 -- 172.21.15.27:0/2333492411 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 6) ==== 59491+0+0 (secure 0 0 0) 0x7f7a84045020 con 0x7f7a8c132ea0 2024-09-21T03:28:28.473 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.470+0000 7f7a897fa640 1 -- 172.21.15.27:0/2333492411 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7f7a84083cf0 con 0x7f7a8c132ea0 2024-09-21T03:28:28.477 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.475+0000 7f7a897fa640 1 -- 172.21.15.27:0/2333492411 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+152684 (secure 0 0 0) 0x7f7a8404c030 con 0x7f7a8c132ea0 2024-09-21T03:28:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.476+0000 7f6ae17f6640 1 -- 172.21.15.27:0/604987024 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6adc12fb20 msgr2=0x7f6adc12ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.476+0000 7f6ae17f6640 1 --2- 172.21.15.27:0/604987024 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6adc12fb20 0x7f6adc12ff30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009be0 tx=0x7f6ad402a770 comp rx=0 tx=0).stop 2024-09-21T03:28:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.476+0000 7f6ae17f6640 1 -- 172.21.15.27:0/604987024 shutdown_connections 2024-09-21T03:28:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.476+0000 7f6ae17f6640 1 --2- 172.21.15.27:0/604987024 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6adc070f60 0x7f6adc071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.476+0000 7f6ae17f6640 1 --2- 172.21.15.27:0/604987024 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6adc12fb20 0x7f6adc12ff30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.476+0000 7f6ae17f6640 1 -- 172.21.15.27:0/604987024 >> 172.21.15.27:0/604987024 conn(0x7f6adc09eb80 msgr2=0x7f6adc09ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:28.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.476+0000 7f6ae17f6640 1 -- 172.21.15.27:0/604987024 shutdown_connections 2024-09-21T03:28:28.479 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.476+0000 7f6ae17f6640 1 -- 172.21.15.27:0/604987024 wait complete. 2024-09-21T03:28:28.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.478+0000 7f6ae17f6640 1 Processor -- start 2024-09-21T03:28:28.480 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.478+0000 7f6ae17f6640 1 -- start start 2024-09-21T03:28:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.479+0000 7f6ae17f6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6adc070f60 0x7f6adc1c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.479+0000 7f6ae17f6640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6adc12fb20 0x7f6adc1c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.479+0000 7f6ae17f6640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6adc130470 0x7f6adc1c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.479+0000 7f6ae17f6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6adc1385e0 con 0x7f6adc070f60 2024-09-21T03:28:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.479+0000 7f6ae17f6640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f6adc138460 con 0x7f6adc12fb20 2024-09-21T03:28:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.479+0000 7f6ae17f6640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f6adc138760 con 0x7f6adc130470 2024-09-21T03:28:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.479+0000 7f6adb7fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6adc130470 0x7f6adc1c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.479+0000 7f6adaffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6adc070f60 0x7f6adc1c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.479+0000 7f6adb7fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6adc130470 0x7f6adc1c6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60654/0 (socket says 172.21.15.27:60654) 2024-09-21T03:28:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.479+0000 7f6adb7fe640 1 -- 172.21.15.27:0/2403071453 learned_addr learned my addr 172.21.15.27:0/2403071453 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.479+0000 7f6ada7fc640 1 --2- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6adc12fb20 0x7f6adc1c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.480+0000 7f6ada7fc640 1 -- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6adc130470 msgr2=0x7f6adc1c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.480+0000 7f6ada7fc640 1 --2- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6adc130470 0x7f6adc1c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.480+0000 7f6ada7fc640 1 -- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6adc070f60 msgr2=0x7f6adc1c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.480+0000 7f6ada7fc640 1 --2- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6adc070f60 0x7f6adc1c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.480+0000 7f6ada7fc640 1 -- 172.21.15.27:0/2403071453 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6adc092750 con 0x7f6adc12fb20 2024-09-21T03:28:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.480+0000 7f6adaffd640 1 --2- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6adc070f60 0x7f6adc1c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.480+0000 7f6ada7fc640 1 --2- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6adc12fb20 0x7f6adc1c6010 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6adc074840 tx=0x7f6acc00cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.480+0000 7f6ac37fe640 1 -- 172.21.15.27:0/2403071453 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc005070 con 0x7f6adc12fb20 2024-09-21T03:28:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.480+0000 7f6ae17f6640 1 -- 172.21.15.27:0/2403071453 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6adc095760 con 0x7f6adc12fb20 2024-09-21T03:28:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.480+0000 7f6ae17f6640 1 -- 172.21.15.27:0/2403071453 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6adc12eb10 con 0x7f6adc12fb20 2024-09-21T03:28:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.481+0000 7f6ac37fe640 1 -- 172.21.15.27:0/2403071453 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6acc005210 con 0x7f6adc12fb20 2024-09-21T03:28:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.481+0000 7f6ac37fe640 1 -- 172.21.15.27:0/2403071453 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc005070 con 0x7f6adc12fb20 2024-09-21T03:28:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.482+0000 7f6ac37fe640 1 -- 172.21.15.27:0/2403071453 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 6) ==== 59491+0+0 (secure 0 0 0) 0x7f6acc00a610 con 0x7f6adc12fb20 2024-09-21T03:28:28.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.482+0000 7f6ae17f6640 1 -- 172.21.15.27:0/2403071453 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6abc005290 con 0x7f6adc12fb20 2024-09-21T03:28:28.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.482+0000 7f6ac37fe640 1 -- 172.21.15.27:0/2403071453 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7f6acc062460 con 0x7f6adc12fb20 2024-09-21T03:28:28.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.490+0000 7f6ac37fe640 1 -- 172.21.15.27:0/2403071453 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+152684 (secure 0 0 0) 0x7f6acc0613a0 con 0x7f6adc12fb20 2024-09-21T03:28:28.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.582+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff10812c510 con 0x7ff10812c100 2024-09-21T03:28:28.585 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.583+0000 7ff0fdffb640 1 -- 172.21.15.27:0/3194146176 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff104064020 con 0x7ff10812c100 2024-09-21T03:28:28.585 INFO:teuthology.orchestra.run.smithi027.stdout:47244640273 2024-09-21T03:28:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.585+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff10812c100 msgr2=0x7ff1081c1810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.585+0000 7ff10f5d2640 1 --2- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff10812c100 0x7ff1081c1810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff10400c7d0 tx=0x7ff10400cc90 comp rx=0 tx=0).stop 2024-09-21T03:28:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.586+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 shutdown_connections 2024-09-21T03:28:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.586+0000 7ff10f5d2640 1 --2- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff108131880 0x7ff1081c2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.586+0000 7ff10f5d2640 1 --2- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff10812cc20 0x7ff1081c1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.586+0000 7ff10f5d2640 1 --2- 172.21.15.27:0/3194146176 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff10812c100 0x7ff1081c1810 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.586+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 >> 172.21.15.27:0/3194146176 conn(0x7ff10809e5e0 msgr2=0x7ff10809a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:28.588 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.586+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 shutdown_connections 2024-09-21T03:28:28.589 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.586+0000 7ff10f5d2640 1 -- 172.21.15.27:0/3194146176 wait complete. 2024-09-21T03:28:28.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.594+0000 7fc169dcb640 1 -- 172.21.15.27:0/3577690611 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc16412cff0 con 0x7fc16412bda0 2024-09-21T03:28:28.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.595+0000 7fc14bfff640 1 -- 172.21.15.27:0/3577690611 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc154061540 con 0x7fc16412bda0 2024-09-21T03:28:28.597 INFO:teuthology.orchestra.run.smithi027.stdout:47244640274 2024-09-21T03:28:28.599 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640274 got 47244640273 for osd.3 2024-09-21T03:28:28.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.598+0000 7fc169dcb640 1 -- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc16412bda0 msgr2=0x7fc1641c2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.598+0000 7fc169dcb640 1 --2- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc16412bda0 0x7fc1641c2100 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc15400c3c0 tx=0x7fc15400c880 comp rx=0 tx=0).stop 2024-09-21T03:28:28.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.598+0000 7fc169dcb640 1 -- 172.21.15.27:0/3577690611 shutdown_connections 2024-09-21T03:28:28.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.598+0000 7fc169dcb640 1 --2- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc16412bda0 0x7fc1641c2100 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.598+0000 7fc169dcb640 1 --2- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc1640734f0 0x7fc1641c1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.598+0000 7fc169dcb640 1 --2- 172.21.15.27:0/3577690611 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc164072ba0 0x7fc1641c1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.598+0000 7fc169dcb640 1 -- 172.21.15.27:0/3577690611 >> 172.21.15.27:0/3577690611 conn(0x7fc164098360 msgr2=0x7fc16409b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:28.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.598+0000 7fc169dcb640 1 -- 172.21.15.27:0/3577690611 shutdown_connections 2024-09-21T03:28:28.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.598+0000 7fc169dcb640 1 -- 172.21.15.27:0/3577690611 wait complete. 2024-09-21T03:28:28.615 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640275 got 47244640274 for osd.0 2024-09-21T03:28:28.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.646+0000 7f6ae17f6640 1 -- 172.21.15.27:0/2403071453 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6abc005580 con 0x7f6adc12fb20 2024-09-21T03:28:28.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.647+0000 7f6ac37fe640 1 -- 172.21.15.27:0/2403071453 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6acc03b400 con 0x7f6adc12fb20 2024-09-21T03:28:28.649 INFO:teuthology.orchestra.run.smithi027.stdout:38654705681 2024-09-21T03:28:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.648+0000 7f6ae17f6640 1 -- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6adc12fb20 msgr2=0x7f6adc1c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.648+0000 7f6ae17f6640 1 --2- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6adc12fb20 0x7f6adc1c6010 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6adc074840 tx=0x7f6acc00cb90 comp rx=0 tx=0).stop 2024-09-21T03:28:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.648+0000 7f6ae17f6640 1 -- 172.21.15.27:0/2403071453 shutdown_connections 2024-09-21T03:28:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.648+0000 7f6ae17f6640 1 --2- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6adc130470 0x7f6adc1c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.648+0000 7f6ae17f6640 1 --2- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6adc12fb20 0x7f6adc1c6010 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.648+0000 7f6ae17f6640 1 --2- 172.21.15.27:0/2403071453 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6adc070f60 0x7f6adc1c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.648+0000 7f6ae17f6640 1 -- 172.21.15.27:0/2403071453 >> 172.21.15.27:0/2403071453 conn(0x7f6adc09eb80 msgr2=0x7f6adc099fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.648+0000 7f6ae17f6640 1 -- 172.21.15.27:0/2403071453 shutdown_connections 2024-09-21T03:28:28.651 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.648+0000 7f6ae17f6640 1 -- 172.21.15.27:0/2403071453 wait complete. 2024-09-21T03:28:28.662 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705683 got 38654705681 for osd.1 2024-09-21T03:28:28.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.694+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f7a6c005580 con 0x7f7a8c132ea0 2024-09-21T03:28:28.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.694+0000 7f7a897fa640 1 -- 172.21.15.27:0/2333492411 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7a8402f070 con 0x7f7a8c132ea0 2024-09-21T03:28:28.696 INFO:teuthology.orchestra.run.smithi027.stdout:47244640273 2024-09-21T03:28:28.698 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.696+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7a8c132ea0 msgr2=0x7f7a8c12f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.696+0000 7f7a92e48640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7a8c132ea0 0x7f7a8c12f1a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7a84009000 tx=0x7f7a84033dd0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.696+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 shutdown_connections 2024-09-21T03:28:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.696+0000 7f7a92e48640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7a8c132ea0 0x7f7a8c12f1a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.696+0000 7f7a92e48640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8c12aaa0 0x7f7a8c12ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.696+0000 7f7a92e48640 1 --2- 172.21.15.27:0/2333492411 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7a8c073c10 0x7f7a8c12e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.696+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 >> 172.21.15.27:0/2333492411 conn(0x7f7a8c09fa20 msgr2=0x7f7a8c1271d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.696+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 shutdown_connections 2024-09-21T03:28:28.699 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:28.696+0000 7f7a92e48640 1 -- 172.21.15.27:0/2333492411 wait complete. 2024-09-21T03:28:28.709 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640274 got 47244640273 for osd.2 2024-09-21T03:28:29.292 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:28:29.289+0000 7f0124833640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-21T03:28:29.292 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:28:29.290+0000 7f0124833640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-21T03:28:29.293 INFO:tasks.ceph.mgr.y.smithi027.stderr:2024-09-21T03:28:29.291+0000 7f0124833640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T03:28:29.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T03:28:29.615 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T03:28:29.663 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T03:28:29.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T03:28:29.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.736+0000 7fabc760a640 1 -- 172.21.15.27:0/101081628 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fabc012a2e0 msgr2=0x7fabc012a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.736+0000 7fabc760a640 1 --2- 172.21.15.27:0/101081628 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fabc012a2e0 0x7fabc012a6f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009900 tx=0x7fabbc02a530 comp rx=0 tx=0).stop 2024-09-21T03:28:29.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.736+0000 7fabc760a640 1 -- 172.21.15.27:0/101081628 shutdown_connections 2024-09-21T03:28:29.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.736+0000 7fabc760a640 1 --2- 172.21.15.27:0/101081628 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fabc012d3d0 0x7fabc012d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.736+0000 7fabc760a640 1 --2- 172.21.15.27:0/101081628 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fabc012a2e0 0x7fabc012a6f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.736+0000 7fabc760a640 1 -- 172.21.15.27:0/101081628 >> 172.21.15.27:0/101081628 conn(0x7fabc009eb80 msgr2=0x7fabc009ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:29.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.737+0000 7fabc760a640 1 -- 172.21.15.27:0/101081628 shutdown_connections 2024-09-21T03:28:29.739 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.737+0000 7fabc760a640 1 -- 172.21.15.27:0/101081628 wait complete. 2024-09-21T03:28:29.740 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.738+0000 7fabc760a640 1 Processor -- start 2024-09-21T03:28:29.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.738+0000 7fabc760a640 1 -- start start 2024-09-21T03:28:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc760a640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fabc012a2e0 0x7fabc0098080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc760a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc012d3d0 0x7fabc00985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc4ba5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc012d3d0 0x7fabc00985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc53a6640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fabc012a2e0 0x7fabc0098080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc4ba5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc012d3d0 0x7fabc00985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56838/0 (socket says 172.21.15.27:56838) 2024-09-21T03:28:29.743 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc4ba5640 1 -- 172.21.15.27:0/1201101703 learned_addr learned my addr 172.21.15.27:0/1201101703 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc53a6640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fabc012a2e0 0x7fabc0098080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60660/0 (socket says 172.21.15.27:60660) 2024-09-21T03:28:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc760a640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fabc012dd20 0x7fabc0098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fabc0136570 con 0x7fabc012d3d0 2024-09-21T03:28:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fabc01363f0 con 0x7fabc012dd20 2024-09-21T03:28:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fabc01366f0 con 0x7fabc012a2e0 2024-09-21T03:28:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc5ba7640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fabc012dd20 0x7fabc0098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc4ba5640 1 -- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fabc012a2e0 msgr2=0x7fabc0098080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc4ba5640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fabc012a2e0 0x7fabc0098080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.740+0000 7fabc4ba5640 1 -- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fabc012dd20 msgr2=0x7fabc0098b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.741+0000 7fabc4ba5640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fabc012dd20 0x7fabc0098b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.741+0000 7fabc4ba5640 1 -- 172.21.15.27:0/1201101703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc01338a0 con 0x7fabc012d3d0 2024-09-21T03:28:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.741+0000 7fabc53a6640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fabc012a2e0 0x7fabc0098080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.741+0000 7fabc5ba7640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fabc012dd20 0x7fabc0098b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.741+0000 7fabc4ba5640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc012d3d0 0x7fabc00985c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fabb000ea70 tx=0x7fabb000ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.741+0000 7fabb5ffb640 1 -- 172.21.15.27:0/1201101703 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb000cde0 con 0x7fabc012d3d0 2024-09-21T03:28:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.741+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc0134420 con 0x7fabc012d3d0 2024-09-21T03:28:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.741+0000 7fabb5ffb640 1 -- 172.21.15.27:0/1201101703 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fabb0004020 con 0x7fabc012d3d0 2024-09-21T03:28:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.741+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc01349c0 con 0x7fabc012d3d0 2024-09-21T03:28:29.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.741+0000 7fabb5ffb640 1 -- 172.21.15.27:0/1201101703 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb0004410 con 0x7fabc012d3d0 2024-09-21T03:28:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.743+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc0132750 con 0x7fabc012d3d0 2024-09-21T03:28:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.743+0000 7fabb5ffb640 1 -- 172.21.15.27:0/1201101703 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) ==== 63336+0+0 (secure 0 0 0) 0x7fabb00035f0 con 0x7fabc012d3d0 2024-09-21T03:28:29.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.744+0000 7fabb5ffb640 1 -- 172.21.15.27:0/1201101703 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7fabb007a640 con 0x7fabc012d3d0 2024-09-21T03:28:29.753 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.751+0000 7fabb5ffb640 1 -- 172.21.15.27:0/1201101703 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fabb00790f0 con 0x7fabc012d3d0 2024-09-21T03:28:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.781+0000 7f5570756640 1 -- 172.21.15.27:0/1445461572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5568094580 msgr2=0x7f5568094990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.781+0000 7f5570756640 1 --2- 172.21.15.27:0/1445461572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5568094580 0x7f5568094990 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5560005fd0 tx=0x7f556002a6e0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.781+0000 7f5570756640 1 -- 172.21.15.27:0/1445461572 shutdown_connections 2024-09-21T03:28:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.781+0000 7f5570756640 1 --2- 172.21.15.27:0/1445461572 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5568094580 0x7f5568094990 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.781+0000 7f5570756640 1 --2- 172.21.15.27:0/1445461572 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f556812dec0 0x7f5568094040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.781+0000 7f5570756640 1 --2- 172.21.15.27:0/1445461572 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f556812d570 0x7f556812d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.781+0000 7f5570756640 1 -- 172.21.15.27:0/1445461572 >> 172.21.15.27:0/1445461572 conn(0x7f556809e5e0 msgr2=0x7f556809aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.782+0000 7f5570756640 1 -- 172.21.15.27:0/1445461572 shutdown_connections 2024-09-21T03:28:29.784 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.782+0000 7f5570756640 1 -- 172.21.15.27:0/1445461572 wait complete. 2024-09-21T03:28:29.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.783+0000 7f5570756640 1 Processor -- start 2024-09-21T03:28:29.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.783+0000 7f5570756640 1 -- start start 2024-09-21T03:28:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.783+0000 7f5570756640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5568094580 0x7f5568124e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.783+0000 7f5570756640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f556812d570 0x7f5568125370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.783+0000 7f5570756640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f556812dec0 0x7f55681258b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f5570756640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5568070770 con 0x7f5568094580 2024-09-21T03:28:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f5570756640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f55680705f0 con 0x7f556812dec0 2024-09-21T03:28:29.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f5570756640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f55680708f0 con 0x7f556812d570 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556dcf1640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f556812d570 0x7f5568125370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556ecf3640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f556812dec0 0x7f55681258b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556e4f2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5568094580 0x7f5568124e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556dcf1640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f556812d570 0x7f5568125370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60668/0 (socket says 172.21.15.27:60668) 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556dcf1640 1 -- 172.21.15.27:0/2798789474 learned_addr learned my addr 172.21.15.27:0/2798789474 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556dcf1640 1 -- 172.21.15.27:0/2798789474 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f556812dec0 msgr2=0x7f55681258b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556dcf1640 1 --2- 172.21.15.27:0/2798789474 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f556812dec0 0x7f55681258b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556dcf1640 1 -- 172.21.15.27:0/2798789474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5568094580 msgr2=0x7f5568124e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556dcf1640 1 --2- 172.21.15.27:0/2798789474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5568094580 0x7f5568124e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556dcf1640 1 -- 172.21.15.27:0/2798789474 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5568127650 con 0x7f556812d570 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556e4f2640 1 --2- 172.21.15.27:0/2798789474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5568094580 0x7f5568124e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.784+0000 7f556dcf1640 1 --2- 172.21.15.27:0/2798789474 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f556812d570 0x7f5568125370 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f555800e9f0 tx=0x7f555800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.785+0000 7f555effd640 1 -- 172.21.15.27:0/2798789474 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555800cd80 con 0x7f556812d570 2024-09-21T03:28:29.788 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.785+0000 7f555effd640 1 -- 172.21.15.27:0/2798789474 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5558004020 con 0x7f556812d570 2024-09-21T03:28:29.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.785+0000 7f555effd640 1 -- 172.21.15.27:0/2798789474 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5558004450 con 0x7f556812d570 2024-09-21T03:28:29.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.785+0000 7f5570756640 1 -- 172.21.15.27:0/2798789474 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5568126f80 con 0x7f556812d570 2024-09-21T03:28:29.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.786+0000 7f5570756640 1 -- 172.21.15.27:0/2798789474 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f55681c7cf0 con 0x7f556812d570 2024-09-21T03:28:29.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.787+0000 7f555effd640 1 -- 172.21.15.27:0/2798789474 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 7) ==== 63336+0+0 (secure 0 0 0) 0x7f55580035f0 con 0x7f556812d570 2024-09-21T03:28:29.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.787+0000 7f555effd640 1 -- 172.21.15.27:0/2798789474 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7f555806e2e0 con 0x7f556812d570 2024-09-21T03:28:29.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.787+0000 7f5570756640 1 -- 172.21.15.27:0/2798789474 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f556812c0c0 con 0x7f556812d570 2024-09-21T03:28:29.793 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.791+0000 7f555effd640 1 -- 172.21.15.27:0/2798789474 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f556812c0c0 con 0x7f556812d570 2024-09-21T03:28:29.824 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.822+0000 7fd9154d6640 1 -- 172.21.15.27:0/2187880792 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd91012a2e0 msgr2=0x7fd91012a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.822+0000 7fd9154d6640 1 --2- 172.21.15.27:0/2187880792 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd91012a2e0 0x7fd91012a6f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd900005fd0 tx=0x7fd90002a540 comp rx=0 tx=0).stop 2024-09-21T03:28:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.822+0000 7fd9154d6640 1 -- 172.21.15.27:0/2187880792 shutdown_connections 2024-09-21T03:28:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.822+0000 7fd9154d6640 1 --2- 172.21.15.27:0/2187880792 >> v2:172.21.15.27:3300/0 conn(0x7fd91012d3d0 0x7fd91012d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.822+0000 7fd9154d6640 1 --2- 172.21.15.27:0/2187880792 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd91012a2e0 0x7fd91012a6f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.822+0000 7fd9154d6640 1 -- 172.21.15.27:0/2187880792 >> 172.21.15.27:0/2187880792 conn(0x7fd91009eb80 msgr2=0x7fd91009ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.822+0000 7fd9154d6640 1 -- 172.21.15.27:0/2187880792 shutdown_connections 2024-09-21T03:28:29.825 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.822+0000 7fd9154d6640 1 -- 172.21.15.27:0/2187880792 wait complete. 2024-09-21T03:28:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.824+0000 7fd9154d6640 1 Processor -- start 2024-09-21T03:28:29.826 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.824+0000 7fd9154d6640 1 -- start start 2024-09-21T03:28:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.825+0000 7fd9154d6640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd91012a2e0 0x7fd9101c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.825+0000 7fd9154d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91012d3d0 0x7fd9101c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.825+0000 7fd9154d6640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd91012dd20 0x7fd9101c3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.825+0000 7fd9154d6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd9101365d0 con 0x7fd91012d3d0 2024-09-21T03:28:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.825+0000 7fd9154d6640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fd910136450 con 0x7fd91012a2e0 2024-09-21T03:28:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.825+0000 7fd9154d6640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fd910136750 con 0x7fd91012dd20 2024-09-21T03:28:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.825+0000 7fd90f7fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd91012dd20 0x7fd9101c3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.825+0000 7fd90effd640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd91012a2e0 0x7fd9101c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.829 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.825+0000 7fd90f7fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd91012dd20 0x7fd9101c3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60676/0 (socket says 172.21.15.27:60676) 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.825+0000 7fd90f7fe640 1 -- 172.21.15.27:0/970288809 learned_addr learned my addr 172.21.15.27:0/970288809 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.825+0000 7fd90effd640 1 --2- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd91012a2e0 0x7fd9101c50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:45150/0 (socket says 172.21.15.27:45150) 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd90effd640 1 -- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd91012dd20 msgr2=0x7fd9101c3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd90effd640 1 --2- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd91012dd20 0x7fd9101c3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd90effd640 1 -- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91012d3d0 msgr2=0x7fd9101c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd90e7fc640 1 --2- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91012d3d0 0x7fd9101c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd90effd640 1 --2- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91012d3d0 0x7fd9101c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd90effd640 1 -- 172.21.15.27:0/970288809 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9101356f0 con 0x7fd91012a2e0 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd90e7fc640 1 --2- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91012d3d0 0x7fd9101c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd90effd640 1 --2- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd91012a2e0 0x7fd9101c50b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd91006ed90 tx=0x7fd9000038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd8f77fe640 1 -- 172.21.15.27:0/970288809 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90003e030 con 0x7fd91012a2e0 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd8f77fe640 1 -- 172.21.15.27:0/970288809 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd900003db0 con 0x7fd91012a2e0 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd9154d6640 1 -- 172.21.15.27:0/970288809 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd910072070 con 0x7fd91012a2e0 2024-09-21T03:28:29.830 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd8f77fe640 1 -- 172.21.15.27:0/970288809 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd900035040 con 0x7fd91012a2e0 2024-09-21T03:28:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.826+0000 7fd9154d6640 1 -- 172.21.15.27:0/970288809 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9101c4240 con 0x7fd91012a2e0 2024-09-21T03:28:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.828+0000 7fd9154d6640 1 -- 172.21.15.27:0/970288809 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd910129770 con 0x7fd91012a2e0 2024-09-21T03:28:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.828+0000 7fd8f77fe640 1 -- 172.21.15.27:0/970288809 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 7) ==== 63336+0+0 (secure 0 0 0) 0x7fd900041030 con 0x7fd91012a2e0 2024-09-21T03:28:29.831 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.829+0000 7fd8f77fe640 1 -- 172.21.15.27:0/970288809 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7fd900089360 con 0x7fd91012a2e0 2024-09-21T03:28:29.838 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.836+0000 7fd8f77fe640 1 -- 172.21.15.27:0/970288809 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fd910129770 con 0x7fd91012a2e0 2024-09-21T03:28:29.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.857+0000 7f357464b640 1 -- 172.21.15.27:0/3524662328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f356c12cf80 msgr2=0x7f356c1294b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.857+0000 7f357464b640 1 --2- 172.21.15.27:0/3524662328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f356c12cf80 0x7f356c1294b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3564005fd0 tx=0x7f356402a8b0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.857+0000 7f357464b640 1 -- 172.21.15.27:0/3524662328 shutdown_connections 2024-09-21T03:28:29.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.857+0000 7f357464b640 1 --2- 172.21.15.27:0/3524662328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f356c12cf80 0x7f356c1294b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.857+0000 7f357464b640 1 --2- 172.21.15.27:0/3524662328 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f356c0727c0 0x7f356c12c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.857+0000 7f357464b640 1 -- 172.21.15.27:0/3524662328 >> 172.21.15.27:0/3524662328 conn(0x7f356c09eb10 msgr2=0x7f356c09ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:29.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.857+0000 7f357464b640 1 -- 172.21.15.27:0/3524662328 shutdown_connections 2024-09-21T03:28:29.860 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.857+0000 7f357464b640 1 -- 172.21.15.27:0/3524662328 wait complete. 2024-09-21T03:28:29.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.859+0000 7f357464b640 1 Processor -- start 2024-09-21T03:28:29.861 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.859+0000 7f357464b640 1 -- start start 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.860+0000 7f357464b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f356c0727c0 0x7f356c096bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.860+0000 7f357464b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f356c12cb70 0x7f356c097110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.860+0000 7f35723e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f356c0727c0 0x7f356c096bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.860+0000 7f35723e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f356c0727c0 0x7f356c096bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56894/0 (socket says 172.21.15.27:56894) 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.860+0000 7f35723e7640 1 -- 172.21.15.27:0/3003602432 learned_addr learned my addr 172.21.15.27:0/3003602432 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.860+0000 7f357464b640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f356c133e50 0x7f356c097650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.860+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f356c06f730 con 0x7f356c0727c0 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.860+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f356c06f5b0 con 0x7f356c133e50 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.860+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f356c06f8b0 con 0x7f356c12cb70 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f35723e7640 1 -- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f356c12cb70 msgr2=0x7f356c097110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.860+0000 7f3571be6640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f356c12cb70 0x7f356c097110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.864 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f35723e7640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f356c12cb70 0x7f356c097110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f3572be8640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f356c133e50 0x7f356c097650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f35723e7640 1 -- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f356c133e50 msgr2=0x7f356c097650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f35723e7640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f356c133e50 0x7f356c097650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f35723e7640 1 -- 172.21.15.27:0/3003602432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f356c12d900 con 0x7f356c0727c0 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f3571be6640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f356c12cb70 0x7f356c097110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f3572be8640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f356c133e50 0x7f356c097650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f35723e7640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f356c0727c0 0x7f356c096bd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f356800e7c0 tx=0x7f356800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f3562ffd640 1 -- 172.21.15.27:0/3003602432 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356800cc60 con 0x7f356c0727c0 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f3562ffd640 1 -- 172.21.15.27:0/3003602432 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f356800ce00 con 0x7f356c0727c0 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f3562ffd640 1 -- 172.21.15.27:0/3003602432 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3568005280 con 0x7f356c0727c0 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.861+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f356c12dcb0 con 0x7f356c0727c0 2024-09-21T03:28:29.865 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.862+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f356c1c7790 con 0x7f356c0727c0 2024-09-21T03:28:29.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.863+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f356c12cf80 con 0x7f356c0727c0 2024-09-21T03:28:29.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.863+0000 7f3562ffd640 1 -- 172.21.15.27:0/3003602432 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 7) ==== 63336+0+0 (secure 0 0 0) 0x7f3568005420 con 0x7f356c0727c0 2024-09-21T03:28:29.866 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.864+0000 7f3562ffd640 1 -- 172.21.15.27:0/3003602432 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7f3568069520 con 0x7f356c0727c0 2024-09-21T03:28:29.869 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.868+0000 7f3562ffd640 1 -- 172.21.15.27:0/3003602432 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f3568068350 con 0x7f356c0727c0 2024-09-21T03:28:29.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.929+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fabc012a6f0 con 0x7fabc012d3d0 2024-09-21T03:28:29.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.930+0000 7fabb5ffb640 1 -- 172.21.15.27:0/1201101703 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fabb0079470 con 0x7fabc012d3d0 2024-09-21T03:28:29.932 INFO:teuthology.orchestra.run.smithi027.stdout:47244640273 2024-09-21T03:28:29.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.932+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 >> v1:172.21.15.27:6800/2405211194 conn(0x7faba4051300 legacy=0x7faba4053810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.932+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc012d3d0 msgr2=0x7fabc00985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.932+0000 7fabc760a640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc012d3d0 0x7fabc00985c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fabb000ea70 tx=0x7fabb000ef30 comp rx=0 tx=0).stop 2024-09-21T03:28:29.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.932+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 shutdown_connections 2024-09-21T03:28:29.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.932+0000 7fabc760a640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fabc012dd20 0x7fabc0098b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.932+0000 7fabc760a640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fabc012d3d0 0x7fabc00985c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.932+0000 7fabc760a640 1 --2- 172.21.15.27:0/1201101703 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fabc012a2e0 0x7fabc0098080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.932+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 >> 172.21.15.27:0/1201101703 conn(0x7fabc009eb80 msgr2=0x7fabc009b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:29.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.932+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 shutdown_connections 2024-09-21T03:28:29.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.932+0000 7fabc760a640 1 -- 172.21.15.27:0/1201101703 wait complete. 2024-09-21T03:28:29.949 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640274 got 47244640273 for osd.3 2024-09-21T03:28:29.974 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.972+0000 7f5570756640 1 -- 172.21.15.27:0/2798789474 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f556812d980 con 0x7f556812d570 2024-09-21T03:28:29.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.973+0000 7f555effd640 1 -- 172.21.15.27:0/2798789474 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5558009460 con 0x7f556812d570 2024-09-21T03:28:29.975 INFO:teuthology.orchestra.run.smithi027.stdout:47244640274 2024-09-21T03:28:29.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.974+0000 7f5570756640 1 -- 172.21.15.27:0/2798789474 >> v1:172.21.15.27:6800/2405211194 conn(0x7f554c051350 legacy=0x7f554c053860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.974+0000 7f5570756640 1 -- 172.21.15.27:0/2798789474 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f556812d570 msgr2=0x7f5568125370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:29.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.974+0000 7f5570756640 1 --2- 172.21.15.27:0/2798789474 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f556812d570 0x7f5568125370 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f555800e9f0 tx=0x7f555800eeb0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.975+0000 7f5570756640 1 -- 172.21.15.27:0/2798789474 shutdown_connections 2024-09-21T03:28:29.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.975+0000 7f5570756640 1 --2- 172.21.15.27:0/2798789474 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f556812dec0 0x7f55681258b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.975+0000 7f5570756640 1 --2- 172.21.15.27:0/2798789474 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f556812d570 0x7f5568125370 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.977 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.975+0000 7f5570756640 1 --2- 172.21.15.27:0/2798789474 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5568094580 0x7f5568124e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:29.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.975+0000 7f5570756640 1 -- 172.21.15.27:0/2798789474 >> 172.21.15.27:0/2798789474 conn(0x7f556809e5e0 msgr2=0x7f55680991e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:29.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.975+0000 7f5570756640 1 -- 172.21.15.27:0/2798789474 shutdown_connections 2024-09-21T03:28:29.978 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:29.975+0000 7f5570756640 1 -- 172.21.15.27:0/2798789474 wait complete. 2024-09-21T03:28:29.988 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640275 got 47244640274 for osd.0 2024-09-21T03:28:30.008 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.006+0000 7fd9154d6640 1 -- 172.21.15.27:0/970288809 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd91012b3f0 con 0x7fd91012a2e0 2024-09-21T03:28:30.009 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.006+0000 7fd8f77fe640 1 -- 172.21.15.27:0/970288809 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd91012b3f0 con 0x7fd91012a2e0 2024-09-21T03:28:30.009 INFO:teuthology.orchestra.run.smithi027.stdout:38654705681 2024-09-21T03:28:30.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.008+0000 7fd9154d6640 1 -- 172.21.15.27:0/970288809 >> v1:172.21.15.27:6800/2405211194 conn(0x7fd8fc051300 legacy=0x7fd8fc053810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:30.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.008+0000 7fd9154d6640 1 -- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd91012a2e0 msgr2=0x7fd9101c50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:30.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.008+0000 7fd9154d6640 1 --2- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd91012a2e0 0x7fd9101c50b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd91006ed90 tx=0x7fd9000038a0 comp rx=0 tx=0).stop 2024-09-21T03:28:30.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.008+0000 7fd9154d6640 1 -- 172.21.15.27:0/970288809 shutdown_connections 2024-09-21T03:28:30.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.008+0000 7fd9154d6640 1 --2- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd91012dd20 0x7fd9101c3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:30.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.008+0000 7fd9154d6640 1 --2- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91012d3d0 0x7fd9101c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:30.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.008+0000 7fd9154d6640 1 --2- 172.21.15.27:0/970288809 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd91012a2e0 0x7fd9101c50b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:30.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.008+0000 7fd9154d6640 1 -- 172.21.15.27:0/970288809 >> 172.21.15.27:0/970288809 conn(0x7fd91009eb80 msgr2=0x7fd91009b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:30.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.008+0000 7fd9154d6640 1 -- 172.21.15.27:0/970288809 shutdown_connections 2024-09-21T03:28:30.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.008+0000 7fd9154d6640 1 -- 172.21.15.27:0/970288809 wait complete. 2024-09-21T03:28:30.021 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705683 got 38654705681 for osd.1 2024-09-21T03:28:30.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.029+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f356c06dda0 con 0x7f356c0727c0 2024-09-21T03:28:30.031 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.029+0000 7f3562ffd640 1 -- 172.21.15.27:0/3003602432 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3568042360 con 0x7f356c0727c0 2024-09-21T03:28:30.032 INFO:teuthology.orchestra.run.smithi027.stdout:47244640273 2024-09-21T03:28:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.031+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 >> v1:172.21.15.27:6800/2405211194 conn(0x7f3550051300 legacy=0x7f3550053810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.031+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f356c0727c0 msgr2=0x7f356c096bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.031+0000 7f357464b640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f356c0727c0 0x7f356c096bd0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f356800e7c0 tx=0x7f356800ec80 comp rx=0 tx=0).stop 2024-09-21T03:28:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.031+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 shutdown_connections 2024-09-21T03:28:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.031+0000 7f357464b640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f356c133e50 0x7f356c097650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.031+0000 7f357464b640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f356c12cb70 0x7f356c097110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.031+0000 7f357464b640 1 --2- 172.21.15.27:0/3003602432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f356c0727c0 0x7f356c096bd0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:30.034 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.031+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 >> 172.21.15.27:0/3003602432 conn(0x7f356c09eb10 msgr2=0x7f356c1239f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.032+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 shutdown_connections 2024-09-21T03:28:30.035 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:30.032+0000 7f357464b640 1 -- 172.21.15.27:0/3003602432 wait complete. 2024-09-21T03:28:30.046 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640274 got 47244640273 for osd.2 2024-09-21T03:28:30.950 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T03:28:30.989 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T03:28:31.023 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T03:28:31.047 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T03:28:31.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.079+0000 7f3358434640 1 -- 172.21.15.27:0/3199226020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33500980f0 msgr2=0x7f3350131a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.079+0000 7f3358434640 1 --2- 172.21.15.27:0/3199226020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33500980f0 0x7f3350131a10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3340005fd0 tx=0x7f334002a4b0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.079+0000 7f3358434640 1 -- 172.21.15.27:0/3199226020 shutdown_connections 2024-09-21T03:28:31.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.079+0000 7f3358434640 1 --2- 172.21.15.27:0/3199226020 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3350131f50 0x7f3350134370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.079+0000 7f3358434640 1 --2- 172.21.15.27:0/3199226020 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33500980f0 0x7f3350131a10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.079+0000 7f3358434640 1 -- 172.21.15.27:0/3199226020 >> 172.21.15.27:0/3199226020 conn(0x7f335009e5e0 msgr2=0x7f335009aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:31.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.079+0000 7f3358434640 1 -- 172.21.15.27:0/3199226020 shutdown_connections 2024-09-21T03:28:31.082 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.079+0000 7f3358434640 1 -- 172.21.15.27:0/3199226020 wait complete. 2024-09-21T03:28:31.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.080+0000 7f3358434640 1 Processor -- start 2024-09-21T03:28:31.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.080+0000 7f3358434640 1 -- start start 2024-09-21T03:28:31.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f3358434640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f33500975d0 0x7f33501c5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f3358434640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33500980f0 0x7f33501c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.084 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f3358434640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3350131f50 0x7f33501c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f3358434640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33501353d0 con 0x7f33500980f0 2024-09-21T03:28:31.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f3358434640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3350135250 con 0x7f33500975d0 2024-09-21T03:28:31.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f3358434640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3350135550 con 0x7f3350131f50 2024-09-21T03:28:31.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f33559cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33500980f0 0x7f33501c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f33569d1640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3350131f50 0x7f33501c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f33561d0640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f33500975d0 0x7f33501c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.085 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f33559cf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33500980f0 0x7f33501c5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56934/0 (socket says 172.21.15.27:56934) 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f33559cf640 1 -- 172.21.15.27:0/3150347261 learned_addr learned my addr 172.21.15.27:0/3150347261 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.081+0000 7f33569d1640 1 --2- 172.21.15.27:0/3150347261 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3350131f50 0x7f33501c63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60748/0 (socket says 172.21.15.27:60748) 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.082+0000 7f33569d1640 1 -- 172.21.15.27:0/3150347261 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f33500975d0 msgr2=0x7f33501c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.082+0000 7f33569d1640 1 --2- 172.21.15.27:0/3150347261 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f33500975d0 0x7f33501c5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.082+0000 7f33569d1640 1 -- 172.21.15.27:0/3150347261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33500980f0 msgr2=0x7f33501c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.082+0000 7f33569d1640 1 --2- 172.21.15.27:0/3150347261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33500980f0 0x7f33501c5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.082+0000 7f33569d1640 1 -- 172.21.15.27:0/3150347261 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3350074630 con 0x7f3350131f50 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.082+0000 7f33569d1640 1 --2- 172.21.15.27:0/3150347261 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3350131f50 0x7f33501c63d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f334800eb50 tx=0x7f334800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.082+0000 7f3346ffd640 1 -- 172.21.15.27:0/3150347261 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f334800ee50 con 0x7f3350131f50 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.082+0000 7f3346ffd640 1 -- 172.21.15.27:0/3150347261 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f334801dae0 con 0x7f3350131f50 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.082+0000 7f3346ffd640 1 -- 172.21.15.27:0/3150347261 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3348003970 con 0x7f3350131f50 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.082+0000 7f3358434640 1 -- 172.21.15.27:0/3150347261 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3350127510 con 0x7f3350131f50 2024-09-21T03:28:31.086 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.082+0000 7f3358434640 1 -- 172.21.15.27:0/3150347261 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3350094040 con 0x7f3350131f50 2024-09-21T03:28:31.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.083+0000 7f3358434640 1 -- 172.21.15.27:0/3150347261 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f335012c980 con 0x7f3350131f50 2024-09-21T03:28:31.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.084+0000 7f3346ffd640 1 -- 172.21.15.27:0/3150347261 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 8) ==== 63442+0+0 (secure 0 0 0) 0x7f3348024030 con 0x7f3350131f50 2024-09-21T03:28:31.087 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.084+0000 7f3346ffd640 1 -- 172.21.15.27:0/3150347261 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7f3348071930 con 0x7f3350131f50 2024-09-21T03:28:31.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.089+0000 7f3346ffd640 1 -- 172.21.15.27:0/3150347261 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f335012c980 con 0x7f3350131f50 2024-09-21T03:28:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.130+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/3505362482 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3b88094750 msgr2=0x7f3b88135b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.130+0000 7f3b8fc7a640 1 --2- 172.21.15.27:0/3505362482 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3b88094750 0x7f3b88135b20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3b80009be0 tx=0x7f3b8002a7c0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.131+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/3505362482 shutdown_connections 2024-09-21T03:28:31.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.131+0000 7f3b8fc7a640 1 --2- 172.21.15.27:0/3505362482 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3b88094750 0x7f3b88135b20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.131+0000 7f3b8fc7a640 1 --2- 172.21.15.27:0/3505362482 >> v2:172.21.15.27:3300/0 conn(0x7f3b8812d570 0x7f3b8812d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.131+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/3505362482 >> 172.21.15.27:0/3505362482 conn(0x7f3b8809e5e0 msgr2=0x7f3b8809aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:31.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.131+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/3505362482 shutdown_connections 2024-09-21T03:28:31.134 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.131+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/3505362482 wait complete. 2024-09-21T03:28:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.133+0000 7f3b8fc7a640 1 Processor -- start 2024-09-21T03:28:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.133+0000 7f3b8fc7a640 1 -- start start 2024-09-21T03:28:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8fc7a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b88094750 0x7f3b881c3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8fc7a640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3b8812d570 0x7f3b881c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8da16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b88094750 0x7f3b881c3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8d215640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3b8812d570 0x7f3b881c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8da16640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b88094750 0x7f3b881c3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56956/0 (socket says 172.21.15.27:56956) 2024-09-21T03:28:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8d215640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3b8812d570 0x7f3b881c3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60774/0 (socket says 172.21.15.27:60774) 2024-09-21T03:28:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8fc7a640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3b8812dec0 0x7f3b881c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8da16640 1 -- 172.21.15.27:0/762733361 learned_addr learned my addr 172.21.15.27:0/762733361 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3b881364b0 con 0x7f3b88094750 2024-09-21T03:28:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3b88136330 con 0x7f3b8812dec0 2024-09-21T03:28:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3b88136630 con 0x7f3b8812d570 2024-09-21T03:28:31.137 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8e217640 1 --2- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3b8812dec0 0x7f3b881c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8da16640 1 -- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3b8812d570 msgr2=0x7f3b881c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8da16640 1 --2- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3b8812d570 0x7f3b881c3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.138 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8da16640 1 -- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3b8812dec0 msgr2=0x7f3b881c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8da16640 1 --2- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3b8812dec0 0x7f3b881c4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8da16640 1 -- 172.21.15.27:0/762733361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b88124550 con 0x7f3b88094750 2024-09-21T03:28:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8d215640 1 --2- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3b8812d570 0x7f3b881c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.134+0000 7f3b8e217640 1 --2- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3b8812dec0 0x7f3b881c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:28:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.135+0000 7f3b8da16640 1 --2- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b88094750 0x7f3b881c3900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400ea20 tx=0x7f3b8400eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.135+0000 7f3b7e7fc640 1 -- 172.21.15.27:0/762733361 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8400cd80 con 0x7f3b88094750 2024-09-21T03:28:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.135+0000 7f3b7e7fc640 1 -- 172.21.15.27:0/762733361 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3b84010430 con 0x7f3b88094750 2024-09-21T03:28:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.135+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b880982b0 con 0x7f3b88094750 2024-09-21T03:28:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.135+0000 7f3b7e7fc640 1 -- 172.21.15.27:0/762733361 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b84010840 con 0x7f3b88094750 2024-09-21T03:28:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.135+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b88134080 con 0x7f3b88094750 2024-09-21T03:28:31.139 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.136+0000 7f3b7e7fc640 1 -- 172.21.15.27:0/762733361 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 8) ==== 63442+0+0 (secure 0 0 0) 0x7f3b840109e0 con 0x7f3b88094750 2024-09-21T03:28:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.137+0000 7f3b7e7fc640 1 -- 172.21.15.27:0/762733361 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7f3b84069bd0 con 0x7f3b88094750 2024-09-21T03:28:31.140 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.137+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b8812d110 con 0x7f3b88094750 2024-09-21T03:28:31.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.141+0000 7f3b7e7fc640 1 -- 172.21.15.27:0/762733361 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f3b8812d110 con 0x7f3b88094750 2024-09-21T03:28:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.169+0000 7fbd42499640 1 -- 172.21.15.27:0/4263418210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd3c1330f0 msgr2=0x7fbd3c135510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.169+0000 7fbd42499640 1 --2- 172.21.15.27:0/4263418210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd3c1330f0 0x7fbd3c135510 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbd34005fd0 tx=0x7fbd3402aaa0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.172 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.170+0000 7fbd42499640 1 -- 172.21.15.27:0/4263418210 shutdown_connections 2024-09-21T03:28:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.170+0000 7fbd42499640 1 --2- 172.21.15.27:0/4263418210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd3c1330f0 0x7fbd3c135510 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.170+0000 7fbd42499640 1 --2- 172.21.15.27:0/4263418210 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbd3c130790 0x7fbd3c132bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.170+0000 7fbd42499640 1 -- 172.21.15.27:0/4263418210 >> 172.21.15.27:0/4263418210 conn(0x7fbd3c09fe40 msgr2=0x7fbd3c0a0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.170+0000 7fbd42499640 1 -- 172.21.15.27:0/4263418210 shutdown_connections 2024-09-21T03:28:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.170+0000 7fbd42499640 1 -- 172.21.15.27:0/4263418210 wait complete. 2024-09-21T03:28:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.171+0000 7fbd42499640 1 Processor -- start 2024-09-21T03:28:31.173 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.171+0000 7fbd42499640 1 -- start start 2024-09-21T03:28:31.174 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd42499640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbd3c12a3f0 0x7fbd3c099690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd42499640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd3c130790 0x7fbd3c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd42499640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbd3c1330f0 0x7fbd3c070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd42499640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3c136420 con 0x7fbd3c130790 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd42499640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fbd3c1362a0 con 0x7fbd3c1330f0 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd42499640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fbd3c1365a0 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd3bfff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbd3c12a3f0 0x7fbd3c099690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd3bfff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbd3c12a3f0 0x7fbd3c099690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60798/0 (socket says 172.21.15.27:60798) 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd40a36640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbd3c1330f0 0x7fbd3c070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd3bfff640 1 -- 172.21.15.27:0/1030375031 learned_addr learned my addr 172.21.15.27:0/1030375031 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd3b7fe640 1 --2- 172.21.15.27:0/1030375031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd3c130790 0x7fbd3c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd3bfff640 1 -- 172.21.15.27:0/1030375031 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbd3c1330f0 msgr2=0x7fbd3c070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd3bfff640 1 --2- 172.21.15.27:0/1030375031 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbd3c1330f0 0x7fbd3c070030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.175 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd3bfff640 1 -- 172.21.15.27:0/1030375031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd3c130790 msgr2=0x7fbd3c06faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd3bfff640 1 --2- 172.21.15.27:0/1030375031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd3c130790 0x7fbd3c06faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.172+0000 7fbd3bfff640 1 -- 172.21.15.27:0/1030375031 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3c12b4a0 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.173+0000 7fbd3bfff640 1 --2- 172.21.15.27:0/1030375031 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbd3c12a3f0 0x7fbd3c099690 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c00c400 tx=0x7fbd2c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.173+0000 7fbd38ff9640 1 -- 172.21.15.27:0/1030375031 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c017070 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.173+0000 7fbd38ff9640 1 -- 172.21.15.27:0/1030375031 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbd2c0053c0 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.173+0000 7fbd42499640 1 -- 172.21.15.27:0/1030375031 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd3c071140 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.173+0000 7fbd38ff9640 1 -- 172.21.15.27:0/1030375031 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c003850 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.173+0000 7fbd42499640 1 -- 172.21.15.27:0/1030375031 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd3c099bd0 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.174+0000 7fbd38ff9640 1 -- 172.21.15.27:0/1030375031 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 8) ==== 63442+0+0 (secure 0 0 0) 0x7fbd2c01a070 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.175+0000 7fbd38ff9640 1 -- 172.21.15.27:0/1030375031 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7fbd2c0683c0 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.175+0000 7fbd42499640 1 -- 172.21.15.27:0/1030375031 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd3c12a800 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.182 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.180+0000 7fbd38ff9640 1 -- 172.21.15.27:0/1030375031 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fbd2c066ca0 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.210+0000 7fb150b38640 1 -- 172.21.15.27:0/734706992 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb14c12df40 msgr2=0x7fb14c1357c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.210+0000 7fb150b38640 1 --2- 172.21.15.27:0/734706992 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb14c12df40 0x7fb14c1357c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb144009be0 tx=0x7fb14402a890 comp rx=0 tx=0).stop 2024-09-21T03:28:31.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.210+0000 7fb150b38640 1 -- 172.21.15.27:0/734706992 shutdown_connections 2024-09-21T03:28:31.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.210+0000 7fb150b38640 1 --2- 172.21.15.27:0/734706992 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb14c12df40 0x7fb14c1357c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.210+0000 7fb150b38640 1 --2- 172.21.15.27:0/734706992 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb14c12a3f0 0x7fb14c12a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.210+0000 7fb150b38640 1 -- 172.21.15.27:0/734706992 >> 172.21.15.27:0/734706992 conn(0x7fb14c09fe40 msgr2=0x7fb14c0a0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:31.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.210+0000 7fb150b38640 1 -- 172.21.15.27:0/734706992 shutdown_connections 2024-09-21T03:28:31.213 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.210+0000 7fb150b38640 1 -- 172.21.15.27:0/734706992 wait complete. 2024-09-21T03:28:31.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.212+0000 7fb150b38640 1 Processor -- start 2024-09-21T03:28:31.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.212+0000 7fb150b38640 1 -- start start 2024-09-21T03:28:31.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.213+0000 7fb150b38640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb14c12a3f0 0x7fb14c131050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.213+0000 7fb150b38640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb14c12d420 0x7fb14c131590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.213+0000 7fb150b38640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb14c12df40 0x7fb14c131ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.213+0000 7fb150b38640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb14c136550 con 0x7fb14c12df40 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.213+0000 7fb150b38640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fb14c1363d0 con 0x7fb14c12a3f0 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.213+0000 7fb150b38640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fb14c1366d0 con 0x7fb14c12d420 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.214+0000 7fb149d9b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb14c12d420 0x7fb14c131590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.214+0000 7fb149d9b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb14c12d420 0x7fb14c131590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60824/0 (socket says 172.21.15.27:60824) 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.214+0000 7fb149d9b640 1 -- 172.21.15.27:0/1307078382 learned_addr learned my addr 172.21.15.27:0/1307078382 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.214+0000 7fb14ad9d640 1 --2- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb14c12df40 0x7fb14c131ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.214+0000 7fb149d9b640 1 -- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb14c12a3f0 msgr2=0x7fb14c131050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.214+0000 7fb14a59c640 1 --2- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb14c12a3f0 0x7fb14c131050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.214+0000 7fb149d9b640 1 --2- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb14c12a3f0 0x7fb14c131050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.214+0000 7fb149d9b640 1 -- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb14c12df40 msgr2=0x7fb14c131ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.214+0000 7fb149d9b640 1 --2- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb14c12df40 0x7fb14c131ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.214+0000 7fb149d9b640 1 -- 172.21.15.27:0/1307078382 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb14c1346b0 con 0x7fb14c12d420 2024-09-21T03:28:31.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.214+0000 7fb14a59c640 1 --2- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb14c12a3f0 0x7fb14c131050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T03:28:31.218 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.215+0000 7fb149d9b640 1 --2- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb14c12d420 0x7fb14c131590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb14000c4c0 tx=0x7fb14000c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.215+0000 7fb13affd640 1 -- 172.21.15.27:0/1307078382 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140016020 con 0x7fb14c12d420 2024-09-21T03:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.215+0000 7fb13affd640 1 -- 172.21.15.27:0/1307078382 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb140012940 con 0x7fb14c12d420 2024-09-21T03:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.215+0000 7fb150b38640 1 -- 172.21.15.27:0/1307078382 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb14c1340e0 con 0x7fb14c12d420 2024-09-21T03:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.215+0000 7fb13affd640 1 -- 172.21.15.27:0/1307078382 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140012d90 con 0x7fb14c12d420 2024-09-21T03:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.215+0000 7fb150b38640 1 -- 172.21.15.27:0/1307078382 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fb14c133a90 con 0x7fb14c12d420 2024-09-21T03:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.217+0000 7fb13affd640 1 -- 172.21.15.27:0/1307078382 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 8) ==== 63442+0+0 (secure 0 0 0) 0x7fb140019020 con 0x7fb14c12d420 2024-09-21T03:28:31.219 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.217+0000 7fb13affd640 1 -- 172.21.15.27:0/1307078382 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7fb140068b90 con 0x7fb14c12d420 2024-09-21T03:28:31.220 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.217+0000 7fb150b38640 1 -- 172.21.15.27:0/1307078382 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb14c12a800 con 0x7fb14c12d420 2024-09-21T03:28:31.223 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.222+0000 7fb13affd640 1 -- 172.21.15.27:0/1307078382 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fb140020030 con 0x7fb14c12d420 2024-09-21T03:28:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.272+0000 7f3358434640 1 -- 172.21.15.27:0/3150347261 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f33500979e0 con 0x7f3350131f50 2024-09-21T03:28:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.273+0000 7f3346ffd640 1 -- 172.21.15.27:0/3150347261 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f334802b060 con 0x7f3350131f50 2024-09-21T03:28:31.276 INFO:teuthology.orchestra.run.smithi027.stdout:47244640275 2024-09-21T03:28:31.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.274+0000 7f3358434640 1 -- 172.21.15.27:0/3150347261 >> v1:172.21.15.27:6800/2405211194 conn(0x7f3334051420 legacy=0x7f3334053930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.274+0000 7f3358434640 1 -- 172.21.15.27:0/3150347261 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3350131f50 msgr2=0x7f33501c63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.274+0000 7f3358434640 1 --2- 172.21.15.27:0/3150347261 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3350131f50 0x7f33501c63d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f334800eb50 tx=0x7f334800c6a0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.275+0000 7f3358434640 1 -- 172.21.15.27:0/3150347261 shutdown_connections 2024-09-21T03:28:31.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.275+0000 7f3358434640 1 --2- 172.21.15.27:0/3150347261 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3350131f50 0x7f33501c63d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.275+0000 7f3358434640 1 --2- 172.21.15.27:0/3150347261 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33500980f0 0x7f33501c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.275+0000 7f3358434640 1 --2- 172.21.15.27:0/3150347261 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f33500975d0 0x7f33501c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.275+0000 7f3358434640 1 -- 172.21.15.27:0/3150347261 >> 172.21.15.27:0/3150347261 conn(0x7f335009e5e0 msgr2=0x7f335012c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:31.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.275+0000 7f3358434640 1 -- 172.21.15.27:0/3150347261 shutdown_connections 2024-09-21T03:28:31.278 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.275+0000 7f3358434640 1 -- 172.21.15.27:0/3150347261 wait complete. 2024-09-21T03:28:31.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640274 got 47244640275 for osd.3 2024-09-21T03:28:31.289 DEBUG:teuthology.parallel:result is None 2024-09-21T03:28:31.315 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.313+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f3b8806a2e0 con 0x7f3b88094750 2024-09-21T03:28:31.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.313+0000 7f3b7e7fc640 1 -- 172.21.15.27:0/762733361 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3b84068c90 con 0x7f3b88094750 2024-09-21T03:28:31.316 INFO:teuthology.orchestra.run.smithi027.stdout:47244640276 2024-09-21T03:28:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.315+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 >> v1:172.21.15.27:6800/2405211194 conn(0x7f3b6c051420 legacy=0x7f3b6c053930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.315+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b88094750 msgr2=0x7f3b881c3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.315+0000 7f3b8fc7a640 1 --2- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b88094750 0x7f3b881c3900 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400ea20 tx=0x7f3b8400eee0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.315+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 shutdown_connections 2024-09-21T03:28:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.315+0000 7f3b8fc7a640 1 --2- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3b8812dec0 0x7f3b881c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.315+0000 7f3b8fc7a640 1 --2- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3b8812d570 0x7f3b881c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.315+0000 7f3b8fc7a640 1 --2- 172.21.15.27:0/762733361 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3b88094750 0x7f3b881c3900 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.315+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 >> 172.21.15.27:0/762733361 conn(0x7f3b8809e5e0 msgr2=0x7f3b88074120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.315+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 shutdown_connections 2024-09-21T03:28:31.318 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.315+0000 7f3b8fc7a640 1 -- 172.21.15.27:0/762733361 wait complete. 2024-09-21T03:28:31.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640275 got 47244640276 for osd.0 2024-09-21T03:28:31.329 DEBUG:teuthology.parallel:result is None 2024-09-21T03:28:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.355+0000 7fbd42499640 1 -- 172.21.15.27:0/1030375031 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fbd3c0691f0 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.355+0000 7fbd38ff9640 1 -- 172.21.15.27:0/1030375031 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbd2c067020 con 0x7fbd3c12a3f0 2024-09-21T03:28:31.357 INFO:teuthology.orchestra.run.smithi027.stdout:38654705684 2024-09-21T03:28:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.357+0000 7fbd42499640 1 -- 172.21.15.27:0/1030375031 >> v1:172.21.15.27:6800/2405211194 conn(0x7fbd28051420 legacy=0x7fbd28053930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.357+0000 7fbd42499640 1 -- 172.21.15.27:0/1030375031 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbd3c12a3f0 msgr2=0x7fbd3c099690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.357+0000 7fbd42499640 1 --2- 172.21.15.27:0/1030375031 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbd3c12a3f0 0x7fbd3c099690 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c00c400 tx=0x7fbd2c00c8c0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.357+0000 7fbd42499640 1 -- 172.21.15.27:0/1030375031 shutdown_connections 2024-09-21T03:28:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.357+0000 7fbd42499640 1 --2- 172.21.15.27:0/1030375031 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbd3c1330f0 0x7fbd3c070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.357+0000 7fbd42499640 1 --2- 172.21.15.27:0/1030375031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbd3c130790 0x7fbd3c06faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.357+0000 7fbd42499640 1 --2- 172.21.15.27:0/1030375031 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbd3c12a3f0 0x7fbd3c099690 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.357+0000 7fbd42499640 1 -- 172.21.15.27:0/1030375031 >> 172.21.15.27:0/1030375031 conn(0x7fbd3c09fe40 msgr2=0x7fbd3c09a420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.357+0000 7fbd42499640 1 -- 172.21.15.27:0/1030375031 shutdown_connections 2024-09-21T03:28:31.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.357+0000 7fbd42499640 1 -- 172.21.15.27:0/1030375031 wait complete. 2024-09-21T03:28:31.370 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705683 got 38654705684 for osd.1 2024-09-21T03:28:31.370 DEBUG:teuthology.parallel:result is None 2024-09-21T03:28:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.386+0000 7fb150b38640 1 -- 172.21.15.27:0/1307078382 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fb14c1261f0 con 0x7fb14c12d420 2024-09-21T03:28:31.388 INFO:teuthology.orchestra.run.smithi027.stdout:47244640275 2024-09-21T03:28:31.388 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.386+0000 7fb13affd640 1 -- 172.21.15.27:0/1307078382 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb1400677f0 con 0x7fb14c12d420 2024-09-21T03:28:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.388+0000 7fb150b38640 1 -- 172.21.15.27:0/1307078382 >> v1:172.21.15.27:6800/2405211194 conn(0x7fb12c051420 legacy=0x7fb12c053930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.388+0000 7fb150b38640 1 -- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb14c12d420 msgr2=0x7fb14c131590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.388+0000 7fb150b38640 1 --2- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb14c12d420 0x7fb14c131590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb14000c4c0 tx=0x7fb14000c980 comp rx=0 tx=0).stop 2024-09-21T03:28:31.390 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.388+0000 7fb150b38640 1 -- 172.21.15.27:0/1307078382 shutdown_connections 2024-09-21T03:28:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.388+0000 7fb150b38640 1 --2- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb14c12df40 0x7fb14c131ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.388+0000 7fb150b38640 1 --2- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb14c12d420 0x7fb14c131590 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.388+0000 7fb150b38640 1 --2- 172.21.15.27:0/1307078382 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb14c12a3f0 0x7fb14c131050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.388+0000 7fb150b38640 1 -- 172.21.15.27:0/1307078382 >> 172.21.15.27:0/1307078382 conn(0x7fb14c09fe40 msgr2=0x7fb14c129250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.388+0000 7fb150b38640 1 -- 172.21.15.27:0/1307078382 shutdown_connections 2024-09-21T03:28:31.391 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.388+0000 7fb150b38640 1 -- 172.21.15.27:0/1307078382 wait complete. 2024-09-21T03:28:31.401 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640274 got 47244640275 for osd.2 2024-09-21T03:28:31.401 DEBUG:teuthology.parallel:result is None 2024-09-21T03:28:31.401 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T03:28:31.401 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T03:28:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.553+0000 7f320f5bc640 1 -- 172.21.15.27:0/1433812632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3208073c10 msgr2=0x7f320812a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.553+0000 7f320f5bc640 1 --2- 172.21.15.27:0/1433812632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3208073c10 0x7f320812a390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3204009830 tx=0x7f320402a480 comp rx=0 tx=0).stop 2024-09-21T03:28:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.553+0000 7f320f5bc640 1 -- 172.21.15.27:0/1433812632 shutdown_connections 2024-09-21T03:28:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.553+0000 7f320f5bc640 1 --2- 172.21.15.27:0/1433812632 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f320812a8d0 0x7f320812d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.556 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.553+0000 7f320f5bc640 1 --2- 172.21.15.27:0/1433812632 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3208073c10 0x7f320812a390 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.553+0000 7f320f5bc640 1 -- 172.21.15.27:0/1433812632 >> 172.21.15.27:0/1433812632 conn(0x7f320809fa20 msgr2=0x7f320809fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.554+0000 7f320f5bc640 1 -- 172.21.15.27:0/1433812632 shutdown_connections 2024-09-21T03:28:31.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.554+0000 7f320f5bc640 1 -- 172.21.15.27:0/1433812632 wait complete. 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320f5bc640 1 Processor -- start 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320f5bc640 1 -- start start 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320f5bc640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3208073c10 0x7f3208096500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320f5bc640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f320812a8d0 0x7f3208096a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320f5bc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f320812ace0 0x7f3208096f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320f5bc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f32081365e0 con 0x7f320812ace0 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320f5bc640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3208136460 con 0x7f320812a8d0 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320f5bc640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3208136760 con 0x7f3208073c10 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320d358640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3208073c10 0x7f3208096500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320cb57640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f320812a8d0 0x7f3208096a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320db59640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f320812ace0 0x7f3208096f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.559 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320d358640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3208073c10 0x7f3208096500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60834/0 (socket says 172.21.15.27:60834) 2024-09-21T03:28:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320d358640 1 -- 172.21.15.27:0/30200679 learned_addr learned my addr 172.21.15.27:0/30200679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.555+0000 7f320cb57640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f320812a8d0 0x7f3208096a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:45240/0 (socket says 172.21.15.27:45240) 2024-09-21T03:28:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.556+0000 7f320cb57640 1 -- 172.21.15.27:0/30200679 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3208073c10 msgr2=0x7f3208096500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.556+0000 7f320cb57640 1 --2- 172.21.15.27:0/30200679 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3208073c10 0x7f3208096500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.556+0000 7f320cb57640 1 -- 172.21.15.27:0/30200679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f320812ace0 msgr2=0x7f3208096f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.556+0000 7f320cb57640 1 --2- 172.21.15.27:0/30200679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f320812ace0 0x7f3208096f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.556+0000 7f320cb57640 1 -- 172.21.15.27:0/30200679 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3208098850 con 0x7f320812a8d0 2024-09-21T03:28:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.556+0000 7f320cb57640 1 --2- 172.21.15.27:0/30200679 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f320812a8d0 0x7f3208096a40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3208073320 tx=0x7f31fc00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:31.560 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.556+0000 7f31f9ffb640 1 -- 172.21.15.27:0/30200679 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc00ccd0 con 0x7f320812a8d0 2024-09-21T03:28:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.556+0000 7f31f9ffb640 1 -- 172.21.15.27:0/30200679 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f31fc004020 con 0x7f320812a8d0 2024-09-21T03:28:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.556+0000 7f320f5bc640 1 -- 172.21.15.27:0/30200679 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32081c3a30 con 0x7f320812a8d0 2024-09-21T03:28:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.556+0000 7f31f9ffb640 1 -- 172.21.15.27:0/30200679 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31fc004410 con 0x7f320812a8d0 2024-09-21T03:28:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.556+0000 7f320f5bc640 1 -- 172.21.15.27:0/30200679 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f32081c3f30 con 0x7f320812a8d0 2024-09-21T03:28:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.557+0000 7f320f5bc640 1 -- 172.21.15.27:0/30200679 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f320809d300 con 0x7f320812a8d0 2024-09-21T03:28:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.558+0000 7f31f9ffb640 1 -- 172.21.15.27:0/30200679 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 8) ==== 63442+0+0 (secure 0 0 0) 0x7f31fc0035f0 con 0x7f320812a8d0 2024-09-21T03:28:31.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.558+0000 7f31f9ffb640 1 -- 172.21.15.27:0/30200679 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7f31fc0698a0 con 0x7f320812a8d0 2024-09-21T03:28:31.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.562+0000 7f31f9ffb640 1 -- 172.21.15.27:0/30200679 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f31fc068180 con 0x7f320812a8d0 2024-09-21T03:28:31.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.718+0000 7f320f5bc640 1 -- 172.21.15.27:0/30200679 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3208099060 con 0x7f31ec051450 2024-09-21T03:28:31.721 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.719+0000 7f31f9ffb640 1 -- 172.21.15.27:0/30200679 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+32044 (unknown 2965378022 0 1009718867) 0x7f3208099060 con 0x7f31ec051450 2024-09-21T03:28:31.722 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:31.722 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-21T03:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.721+0000 7f320f5bc640 1 -- 172.21.15.27:0/30200679 >> v1:172.21.15.27:6800/2405211194 conn(0x7f31ec051450 legacy=0x7f31ec053960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.721+0000 7f320f5bc640 1 -- 172.21.15.27:0/30200679 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f320812a8d0 msgr2=0x7f3208096a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.721+0000 7f320f5bc640 1 --2- 172.21.15.27:0/30200679 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f320812a8d0 0x7f3208096a40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3208073320 tx=0x7f31fc00ee50 comp rx=0 tx=0).stop 2024-09-21T03:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.721+0000 7f320f5bc640 1 -- 172.21.15.27:0/30200679 shutdown_connections 2024-09-21T03:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.721+0000 7f320f5bc640 1 --2- 172.21.15.27:0/30200679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f320812ace0 0x7f3208096f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.721+0000 7f320f5bc640 1 --2- 172.21.15.27:0/30200679 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f320812a8d0 0x7f3208096a40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.721+0000 7f320f5bc640 1 --2- 172.21.15.27:0/30200679 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3208073c10 0x7f3208096500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.721+0000 7f320f5bc640 1 -- 172.21.15.27:0/30200679 >> 172.21.15.27:0/30200679 conn(0x7f320809fa20 msgr2=0x7f3208128dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.722+0000 7f320f5bc640 1 -- 172.21.15.27:0/30200679 shutdown_connections 2024-09-21T03:28:31.725 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.722+0000 7f320f5bc640 1 -- 172.21.15.27:0/30200679 wait complete. 2024-09-21T03:28:31.735 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-09-21T03:28:30.295724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":114144,"kb_used_data":1872,"kb_used_omap":6,"kb_used_meta":112185,"kb_avail":377373216,"statfs":{"total":386547056640,"available":386430173184,"internally_reserved":0,"allocated":1916928,"data_stored":1391558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":114878231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.003558"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.286112+0000","last_change":"2024-09-21T03:27:15.282474+0000","last_active":"2024-09-21T03:28:28.286112+0000","last_peered":"2024-09-21T03:28:28.286112+0000","last_clean":"2024-09-21T03:28:28.286112+0000","last_became_active":"2024-09-21T03:27:13.273870+0000","last_became_peered":"2024-09-21T03:27:13.273870+0000","last_unstale":"2024-09-21T03:28:28.286112+0000","last_undegraded":"2024-09-21T03:28:28.286112+0000","last_fullsized":"2024-09-21T03:28:28.286112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:22.300880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393285,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.289389+0000","last_change":"2024-09-21T03:27:15.287459+0000","last_active":"2024-09-21T03:28:28.289389+0000","last_peered":"2024-09-21T03:28:28.289389+0000","last_clean":"2024-09-21T03:28:28.289389+0000","last_became_active":"2024-09-21T03:27:13.271654+0000","last_became_peered":"2024-09-21T03:27:13.271654+0000","last_unstale":"2024-09-21T03:28:28.289389+0000","last_undegraded":"2024-09-21T03:28:28.289389+0000","last_fullsized":"2024-09-21T03:28:28.289389+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:45:52.098322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044937200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.286045+0000","last_change":"2024-09-21T03:27:15.282111+0000","last_active":"2024-09-21T03:28:28.286045+0000","last_peered":"2024-09-21T03:28:28.286045+0000","last_clean":"2024-09-21T03:28:28.286045+0000","last_became_active":"2024-09-21T03:27:13.273857+0000","last_became_peered":"2024-09-21T03:27:13.273857+0000","last_unstale":"2024-09-21T03:28:28.286045+0000","last_undegraded":"2024-09-21T03:28:28.286045+0000","last_fullsized":"2024-09-21T03:28:28.286045+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:47:49.923357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.289477+0000","last_change":"2024-09-21T03:27:15.287512+0000","last_active":"2024-09-21T03:28:28.289477+0000","last_peered":"2024-09-21T03:28:28.289477+0000","last_clean":"2024-09-21T03:28:28.289477+0000","last_became_active":"2024-09-21T03:27:13.271520+0000","last_became_peered":"2024-09-21T03:27:13.271520+0000","last_unstale":"2024-09-21T03:28:28.289477+0000","last_undegraded":"2024-09-21T03:28:28.289477+0000","last_fullsized":"2024-09-21T03:28:28.289477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:54.936725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049584300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.285997+0000","last_change":"2024-09-21T03:27:15.282023+0000","last_active":"2024-09-21T03:28:28.285997+0000","last_peered":"2024-09-21T03:28:28.285997+0000","last_clean":"2024-09-21T03:28:28.285997+0000","last_became_active":"2024-09-21T03:27:13.276424+0000","last_became_peered":"2024-09-21T03:27:13.276424+0000","last_unstale":"2024-09-21T03:28:28.285997+0000","last_undegraded":"2024-09-21T03:28:28.285997+0000","last_fullsized":"2024-09-21T03:28:28.285997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:23.923919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010094349999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.344069+0000","last_change":"2024-09-21T03:27:15.273538+0000","last_active":"2024-09-21T03:28:28.344069+0000","last_peered":"2024-09-21T03:28:28.344069+0000","last_clean":"2024-09-21T03:28:28.344069+0000","last_became_active":"2024-09-21T03:27:13.273714+0000","last_became_peered":"2024-09-21T03:27:13.273714+0000","last_unstale":"2024-09-21T03:28:28.344069+0000","last_undegraded":"2024-09-21T03:28:28.344069+0000","last_fullsized":"2024-09-21T03:28:28.344069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:31:40.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056217599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.285921+0000","last_change":"2024-09-21T03:27:15.281443+0000","last_active":"2024-09-21T03:28:28.285921+0000","last_peered":"2024-09-21T03:28:28.285921+0000","last_clean":"2024-09-21T03:28:28.285921+0000","last_became_active":"2024-09-21T03:27:13.273967+0000","last_became_peered":"2024-09-21T03:27:13.273967+0000","last_unstale":"2024-09-21T03:28:28.285921+0000","last_undegraded":"2024-09-21T03:28:28.285921+0000","last_fullsized":"2024-09-21T03:28:28.285921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:53.091651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045280299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":26,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.285959+0000","last_change":"2024-09-21T03:27:15.282372+0000","last_active":"2024-09-21T03:28:28.285959+0000","last_peered":"2024-09-21T03:28:28.285959+0000","last_clean":"2024-09-21T03:28:28.285959+0000","last_became_active":"2024-09-21T03:27:13.276509+0000","last_became_peered":"2024-09-21T03:27:13.276509+0000","last_unstale":"2024-09-21T03:28:28.285959+0000","last_undegraded":"2024-09-21T03:28:28.285959+0000","last_fullsized":"2024-09-21T03:28:28.285959+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:22:49.703014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013599459999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":153,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.289478+0000","last_change":"2024-09-21T03:27:09.247989+0000","last_active":"2024-09-21T03:28:28.289478+0000","last_peered":"2024-09-21T03:28:28.289478+0000","last_clean":"2024-09-21T03:28:28.289478+0000","last_became_active":"2024-09-21T03:27:09.247627+0000","last_became_peered":"2024-09-21T03:27:09.247627+0000","last_unstale":"2024-09-21T03:28:28.289478+0000","last_undegraded":"2024-09-21T03:28:28.289478+0000","last_fullsized":"2024-09-21T03:28:28.289478+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_clean_scrub_stamp":"2024-09-21T03:27:08.233475+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:30:49.675995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640275,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":52715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640275,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":52696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705684,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":778240,"data_stored":643083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640276,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":774144,"data_stored":643064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T03:28:31.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T03:28:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.847+0000 7f64cfd26640 1 -- 172.21.15.27:0/386643639 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f64c812f900 msgr2=0x7f64c81349b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.847+0000 7f64cfd26640 1 --2- 172.21.15.27:0/386643639 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f64c812f900 0x7f64c81349b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f64bc009900 tx=0x7f64bc02a490 comp rx=0 tx=0).stop 2024-09-21T03:28:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.848+0000 7f64cfd26640 1 -- 172.21.15.27:0/386643639 shutdown_connections 2024-09-21T03:28:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.848+0000 7f64cfd26640 1 --2- 172.21.15.27:0/386643639 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f64c8130280 0x7f64c8136f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.848+0000 7f64cfd26640 1 --2- 172.21.15.27:0/386643639 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f64c812f900 0x7f64c81349b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.850 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.848+0000 7f64cfd26640 1 -- 172.21.15.27:0/386643639 >> 172.21.15.27:0/386643639 conn(0x7f64c80a2050 msgr2=0x7f64c80a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.848+0000 7f64cfd26640 1 -- 172.21.15.27:0/386643639 shutdown_connections 2024-09-21T03:28:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.848+0000 7f64cfd26640 1 -- 172.21.15.27:0/386643639 wait complete. 2024-09-21T03:28:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.849+0000 7f64cfd26640 1 Processor -- start 2024-09-21T03:28:31.851 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.849+0000 7f64cfd26640 1 -- start start 2024-09-21T03:28:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.849+0000 7f64cfd26640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f64c8071a30 0x7f64c81c5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.849+0000 7f64cfd26640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f64c812f900 0x7f64c81c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.849+0000 7f64cfd26640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c8130280 0x7f64c81c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.849+0000 7f64cfd26640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f64c81381d0 con 0x7f64c8130280 2024-09-21T03:28:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.849+0000 7f64cfd26640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f64c8138050 con 0x7f64c812f900 2024-09-21T03:28:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.849+0000 7f64cfd26640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f64c8138920 con 0x7f64c8071a30 2024-09-21T03:28:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.849+0000 7f64ce2c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c8130280 0x7f64c81c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.853 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.849+0000 7f64cd2c1640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f64c812f900 0x7f64c81c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.849+0000 7f64cdac2640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f64c8071a30 0x7f64c81c5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cd2c1640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f64c812f900 0x7f64c81c6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:45256/0 (socket says 172.21.15.27:45256) 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64ce2c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c8130280 0x7f64c81c6550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:57014/0 (socket says 172.21.15.27:57014) 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cdac2640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f64c8071a30 0x7f64c81c5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:60840/0 (socket says 172.21.15.27:60840) 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cd2c1640 1 -- 172.21.15.27:0/1906156274 learned_addr learned my addr 172.21.15.27:0/1906156274 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cd2c1640 1 -- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f64c8071a30 msgr2=0x7f64c81c5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cd2c1640 1 --2- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f64c8071a30 0x7f64c81c5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cd2c1640 1 -- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c8130280 msgr2=0x7f64c81c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cd2c1640 1 --2- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c8130280 0x7f64c81c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cd2c1640 1 -- 172.21.15.27:0/1906156274 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c81339f0 con 0x7f64c812f900 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64ce2c3640 1 --2- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c8130280 0x7f64c81c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cdac2640 1 --2- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f64c8071a30 0x7f64c81c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:31.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cd2c1640 1 --2- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f64c812f900 0x7f64c81c6010 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f64bc000c00 tx=0x7f64bc0038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:31.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64ba7fc640 1 -- 172.21.15.27:0/1906156274 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc03f070 con 0x7f64c812f900 2024-09-21T03:28:31.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64ba7fc640 1 -- 172.21.15.27:0/1906156274 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f64bc0047a0 con 0x7f64c812f900 2024-09-21T03:28:31.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cfd26640 1 -- 172.21.15.27:0/1906156274 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64c8133440 con 0x7f64c812f900 2024-09-21T03:28:31.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64ba7fc640 1 -- 172.21.15.27:0/1906156274 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc035030 con 0x7f64c812f900 2024-09-21T03:28:31.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.850+0000 7f64cfd26640 1 -- 172.21.15.27:0/1906156274 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64c809b660 con 0x7f64c812f900 2024-09-21T03:28:31.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.851+0000 7f64cfd26640 1 -- 172.21.15.27:0/1906156274 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64c8070cb0 con 0x7f64c812f900 2024-09-21T03:28:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.852+0000 7f64ba7fc640 1 -- 172.21.15.27:0/1906156274 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 8) ==== 63442+0+0 (secure 0 0 0) 0x7f64bc042070 con 0x7f64c812f900 2024-09-21T03:28:31.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.852+0000 7f64ba7fc640 1 -- 172.21.15.27:0/1906156274 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7f64bc089d10 con 0x7f64c812f900 2024-09-21T03:28:31.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:31.856+0000 7f64ba7fc640 1 -- 172.21.15.27:0/1906156274 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f64bc088970 con 0x7f64c812f900 2024-09-21T03:28:32.013 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.011+0000 7f64cfd26640 1 -- 172.21.15.27:0/1906156274 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f64c8132fd0 con 0x7f64ac051420 2024-09-21T03:28:32.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.011+0000 7f64ba7fc640 1 -- 172.21.15.27:0/1906156274 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+32044 (unknown 2965378022 0 1009718867) 0x7f64c8132fd0 con 0x7f64ac051420 2024-09-21T03:28:32.014 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:32.014 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-21T03:28:32.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.013+0000 7f64cfd26640 1 -- 172.21.15.27:0/1906156274 >> v1:172.21.15.27:6800/2405211194 conn(0x7f64ac051420 legacy=0x7f64ac053930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.013+0000 7f64cfd26640 1 -- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f64c812f900 msgr2=0x7f64c81c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.013+0000 7f64cfd26640 1 --2- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f64c812f900 0x7f64c81c6010 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f64bc000c00 tx=0x7f64bc0038a0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.013+0000 7f64cfd26640 1 -- 172.21.15.27:0/1906156274 shutdown_connections 2024-09-21T03:28:32.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.013+0000 7f64cfd26640 1 --2- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f64c8130280 0x7f64c81c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.013+0000 7f64cfd26640 1 --2- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f64c812f900 0x7f64c81c6010 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.013+0000 7f64cfd26640 1 --2- 172.21.15.27:0/1906156274 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f64c8071a30 0x7f64c81c5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.013+0000 7f64cfd26640 1 -- 172.21.15.27:0/1906156274 >> 172.21.15.27:0/1906156274 conn(0x7f64c80a2050 msgr2=0x7f64c81369b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:32.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.013+0000 7f64cfd26640 1 -- 172.21.15.27:0/1906156274 shutdown_connections 2024-09-21T03:28:32.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.013+0000 7f64cfd26640 1 -- 172.21.15.27:0/1906156274 wait complete. 2024-09-21T03:28:32.026 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-09-21T03:28:30.295724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":114144,"kb_used_data":1872,"kb_used_omap":6,"kb_used_meta":112185,"kb_avail":377373216,"statfs":{"total":386547056640,"available":386430173184,"internally_reserved":0,"allocated":1916928,"data_stored":1391558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":114878231},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.003558"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.286112+0000","last_change":"2024-09-21T03:27:15.282474+0000","last_active":"2024-09-21T03:28:28.286112+0000","last_peered":"2024-09-21T03:28:28.286112+0000","last_clean":"2024-09-21T03:28:28.286112+0000","last_became_active":"2024-09-21T03:27:13.273870+0000","last_became_peered":"2024-09-21T03:27:13.273870+0000","last_unstale":"2024-09-21T03:28:28.286112+0000","last_undegraded":"2024-09-21T03:28:28.286112+0000","last_fullsized":"2024-09-21T03:28:28.286112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:22.300880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393285,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.289389+0000","last_change":"2024-09-21T03:27:15.287459+0000","last_active":"2024-09-21T03:28:28.289389+0000","last_peered":"2024-09-21T03:28:28.289389+0000","last_clean":"2024-09-21T03:28:28.289389+0000","last_became_active":"2024-09-21T03:27:13.271654+0000","last_became_peered":"2024-09-21T03:27:13.271654+0000","last_unstale":"2024-09-21T03:28:28.289389+0000","last_undegraded":"2024-09-21T03:28:28.289389+0000","last_fullsized":"2024-09-21T03:28:28.289389+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:45:52.098322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044937200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.286045+0000","last_change":"2024-09-21T03:27:15.282111+0000","last_active":"2024-09-21T03:28:28.286045+0000","last_peered":"2024-09-21T03:28:28.286045+0000","last_clean":"2024-09-21T03:28:28.286045+0000","last_became_active":"2024-09-21T03:27:13.273857+0000","last_became_peered":"2024-09-21T03:27:13.273857+0000","last_unstale":"2024-09-21T03:28:28.286045+0000","last_undegraded":"2024-09-21T03:28:28.286045+0000","last_fullsized":"2024-09-21T03:28:28.286045+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:47:49.923357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.289477+0000","last_change":"2024-09-21T03:27:15.287512+0000","last_active":"2024-09-21T03:28:28.289477+0000","last_peered":"2024-09-21T03:28:28.289477+0000","last_clean":"2024-09-21T03:28:28.289477+0000","last_became_active":"2024-09-21T03:27:13.271520+0000","last_became_peered":"2024-09-21T03:27:13.271520+0000","last_unstale":"2024-09-21T03:28:28.289477+0000","last_undegraded":"2024-09-21T03:28:28.289477+0000","last_fullsized":"2024-09-21T03:28:28.289477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:54.936725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049584300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.285997+0000","last_change":"2024-09-21T03:27:15.282023+0000","last_active":"2024-09-21T03:28:28.285997+0000","last_peered":"2024-09-21T03:28:28.285997+0000","last_clean":"2024-09-21T03:28:28.285997+0000","last_became_active":"2024-09-21T03:27:13.276424+0000","last_became_peered":"2024-09-21T03:27:13.276424+0000","last_unstale":"2024-09-21T03:28:28.285997+0000","last_undegraded":"2024-09-21T03:28:28.285997+0000","last_fullsized":"2024-09-21T03:28:28.285997+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:23.923919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010094349999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.344069+0000","last_change":"2024-09-21T03:27:15.273538+0000","last_active":"2024-09-21T03:28:28.344069+0000","last_peered":"2024-09-21T03:28:28.344069+0000","last_clean":"2024-09-21T03:28:28.344069+0000","last_became_active":"2024-09-21T03:27:13.273714+0000","last_became_peered":"2024-09-21T03:27:13.273714+0000","last_unstale":"2024-09-21T03:28:28.344069+0000","last_undegraded":"2024-09-21T03:28:28.344069+0000","last_fullsized":"2024-09-21T03:28:28.344069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:31:40.805997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056217599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.285921+0000","last_change":"2024-09-21T03:27:15.281443+0000","last_active":"2024-09-21T03:28:28.285921+0000","last_peered":"2024-09-21T03:28:28.285921+0000","last_clean":"2024-09-21T03:28:28.285921+0000","last_became_active":"2024-09-21T03:27:13.273967+0000","last_became_peered":"2024-09-21T03:27:13.273967+0000","last_unstale":"2024-09-21T03:28:28.285921+0000","last_undegraded":"2024-09-21T03:28:28.285921+0000","last_fullsized":"2024-09-21T03:28:28.285921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:53.091651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045280299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":26,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.285959+0000","last_change":"2024-09-21T03:27:15.282372+0000","last_active":"2024-09-21T03:28:28.285959+0000","last_peered":"2024-09-21T03:28:28.285959+0000","last_clean":"2024-09-21T03:28:28.285959+0000","last_became_active":"2024-09-21T03:27:13.276509+0000","last_became_peered":"2024-09-21T03:27:13.276509+0000","last_unstale":"2024-09-21T03:28:28.285959+0000","last_undegraded":"2024-09-21T03:28:28.285959+0000","last_fullsized":"2024-09-21T03:28:28.285959+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:22:49.703014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013599459999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":153,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-09-21T03:28:28.289478+0000","last_change":"2024-09-21T03:27:09.247989+0000","last_active":"2024-09-21T03:28:28.289478+0000","last_peered":"2024-09-21T03:28:28.289478+0000","last_clean":"2024-09-21T03:28:28.289478+0000","last_became_active":"2024-09-21T03:27:09.247627+0000","last_became_peered":"2024-09-21T03:27:09.247627+0000","last_unstale":"2024-09-21T03:28:28.289478+0000","last_undegraded":"2024-09-21T03:28:28.289478+0000","last_fullsized":"2024-09-21T03:28:28.289478+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_clean_scrub_stamp":"2024-09-21T03:27:08.233475+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:30:49.675995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640275,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":52715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640275,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":52696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705684,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":778240,"data_stored":643083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640276,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":774144,"data_stored":643064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T03:28:32.028 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T03:28:32.028 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T03:28:32.028 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T03:28:32.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T03:28:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.139+0000 7fe617346640 1 -- 172.21.15.27:0/434050928 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe610071a30 msgr2=0x7fe610070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.139+0000 7fe617346640 1 --2- 172.21.15.27:0/434050928 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe610071a30 0x7fe610070cb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009830 tx=0x7fe60c02a3f0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.139+0000 7fe617346640 1 -- 172.21.15.27:0/434050928 shutdown_connections 2024-09-21T03:28:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.139+0000 7fe617346640 1 --2- 172.21.15.27:0/434050928 >> v2:172.21.15.27:3300/0 conn(0x7fe61012f900 0x7fe61012c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.139+0000 7fe617346640 1 --2- 172.21.15.27:0/434050928 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe610071a30 0x7fe610070cb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.139+0000 7fe617346640 1 -- 172.21.15.27:0/434050928 >> 172.21.15.27:0/434050928 conn(0x7fe6100a2050 msgr2=0x7fe6100a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:32.142 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.140+0000 7fe617346640 1 -- 172.21.15.27:0/434050928 shutdown_connections 2024-09-21T03:28:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.140+0000 7fe617346640 1 -- 172.21.15.27:0/434050928 wait complete. 2024-09-21T03:28:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.141+0000 7fe617346640 1 Processor -- start 2024-09-21T03:28:32.143 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.141+0000 7fe617346640 1 -- start start 2024-09-21T03:28:32.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.141+0000 7fe617346640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe610071a30 0x7fe6101309a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:32.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.141+0000 7fe617346640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe61012f900 0x7fe610130ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:32.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.141+0000 7fe617346640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe610130280 0x7fe610131420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:32.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.141+0000 7fe617346640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fe610138490 con 0x7fe610130280 2024-09-21T03:28:32.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.141+0000 7fe617346640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fe610138310 con 0x7fe610071a30 2024-09-21T03:28:32.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.141+0000 7fe617346640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fe610138610 con 0x7fe61012f900 2024-09-21T03:28:32.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6150e2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe610071a30 0x7fe6101309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:32.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6150e2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe610071a30 0x7fe6101309a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:45274/0 (socket says 172.21.15.27:45274) 2024-09-21T03:28:32.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6158e3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe610130280 0x7fe610131420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:32.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6150e2640 1 -- 172.21.15.27:0/3698006912 learned_addr learned my addr 172.21.15.27:0/3698006912 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:32.145 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6148e1640 1 --2- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe61012f900 0x7fe610130ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6150e2640 1 -- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe61012f900 msgr2=0x7fe610130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6150e2640 1 --2- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe61012f900 0x7fe610130ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6150e2640 1 -- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe610130280 msgr2=0x7fe610131420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6150e2640 1 --2- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe610130280 0x7fe610131420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6150e2640 1 -- 172.21.15.27:0/3698006912 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6101329c0 con 0x7fe610071a30 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6148e1640 1 --2- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe61012f900 0x7fe610130ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6158e3640 1 --2- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe610130280 0x7fe610131420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe6150e2640 1 --2- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe610071a30 0x7fe6101309a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009950 tx=0x7fe60c033c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe605ffb640 1 -- 172.21.15.27:0/3698006912 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c005c90 con 0x7fe610071a30 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe605ffb640 1 -- 172.21.15.27:0/3698006912 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fe60c031e50 con 0x7fe610071a30 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe617346640 1 -- 172.21.15.27:0/3698006912 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe610133900 con 0x7fe610071a30 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.142+0000 7fe605ffb640 1 -- 172.21.15.27:0/3698006912 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe60c005260 con 0x7fe610071a30 2024-09-21T03:28:32.146 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.143+0000 7fe617346640 1 -- 172.21.15.27:0/3698006912 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6101c5a60 con 0x7fe610071a30 2024-09-21T03:28:32.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.144+0000 7fe617346640 1 -- 172.21.15.27:0/3698006912 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe61012aeb0 con 0x7fe610071a30 2024-09-21T03:28:32.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.144+0000 7fe605ffb640 1 -- 172.21.15.27:0/3698006912 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 8) ==== 63442+0+0 (secure 0 0 0) 0x7fe60c005400 con 0x7fe610071a30 2024-09-21T03:28:32.147 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.144+0000 7fe605ffb640 1 -- 172.21.15.27:0/3698006912 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7fe60c089f30 con 0x7fe610071a30 2024-09-21T03:28:32.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.148+0000 7fe605ffb640 1 -- 172.21.15.27:0/3698006912 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fe61012aeb0 con 0x7fe610071a30 2024-09-21T03:28:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.331+0000 7fe617346640 1 -- 172.21.15.27:0/3698006912 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe610070cb0 con 0x7fe610071a30 2024-09-21T03:28:32.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.331+0000 7fe605ffb640 1 -- 172.21.15.27:0/3698006912 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fe60c03f030 con 0x7fe610071a30 2024-09-21T03:28:32.333 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:32.334 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T03:28:32.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.333+0000 7fe617346640 1 -- 172.21.15.27:0/3698006912 >> v1:172.21.15.27:6800/2405211194 conn(0x7fe5f4051420 legacy=0x7fe5f4053930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.333+0000 7fe617346640 1 -- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe610071a30 msgr2=0x7fe6101309a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.333+0000 7fe617346640 1 --2- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe610071a30 0x7fe6101309a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009950 tx=0x7fe60c033c20 comp rx=0 tx=0).stop 2024-09-21T03:28:32.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.333+0000 7fe617346640 1 -- 172.21.15.27:0/3698006912 shutdown_connections 2024-09-21T03:28:32.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.333+0000 7fe617346640 1 --2- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe610130280 0x7fe610131420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.333+0000 7fe617346640 1 --2- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fe61012f900 0x7fe610130ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.333+0000 7fe617346640 1 --2- 172.21.15.27:0/3698006912 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fe610071a30 0x7fe6101309a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.333+0000 7fe617346640 1 -- 172.21.15.27:0/3698006912 >> 172.21.15.27:0/3698006912 conn(0x7fe6100a2050 msgr2=0x7fe61009b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:32.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.333+0000 7fe617346640 1 -- 172.21.15.27:0/3698006912 shutdown_connections 2024-09-21T03:28:32.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.333+0000 7fe617346640 1 -- 172.21.15.27:0/3698006912 wait complete. 2024-09-21T03:28:32.346 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T03:28:32.347 INFO:teuthology.run_tasks:Running task ceph.restart... 2024-09-21T03:28:32.360 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-21T03:28:32.360 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:28:32.385 INFO:tasks.ceph.osd.0:Stopped 2024-09-21T03:28:32.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-09-21T03:28:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.539+0000 7f0ad598c640 1 -- 172.21.15.27:0/867595763 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0ad012fc70 msgr2=0x7f0ad0130080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.540+0000 7f0ad598c640 1 --2- 172.21.15.27:0/867595763 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0ad012fc70 0x7f0ad0130080 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009830 tx=0x7f0ac002a3f0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.542 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.540+0000 7f0ad598c640 1 -- 172.21.15.27:0/867595763 shutdown_connections 2024-09-21T03:28:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.540+0000 7f0ad598c640 1 --2- 172.21.15.27:0/867595763 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0ad0072d70 0x7f0ad0073180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.540+0000 7f0ad598c640 1 --2- 172.21.15.27:0/867595763 >> v2:172.21.15.27:3300/0 conn(0x7f0ad01305c0 0x7f0ad0072830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.540+0000 7f0ad598c640 1 --2- 172.21.15.27:0/867595763 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0ad012fc70 0x7f0ad0130080 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.540+0000 7f0ad598c640 1 -- 172.21.15.27:0/867595763 >> 172.21.15.27:0/867595763 conn(0x7f0ad00a1f40 msgr2=0x7f0ad00a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.540+0000 7f0ad598c640 1 -- 172.21.15.27:0/867595763 shutdown_connections 2024-09-21T03:28:32.543 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.540+0000 7f0ad598c640 1 -- 172.21.15.27:0/867595763 wait complete. 2024-09-21T03:28:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.541+0000 7f0ad598c640 1 Processor -- start 2024-09-21T03:28:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.541+0000 7f0ad598c640 1 -- start start 2024-09-21T03:28:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.541+0000 7f0ad598c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0ad0072d70 0x7f0ad009b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ad598c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0ad012fc70 0x7f0ad009b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ad598c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad01305c0 0x7f0ad009bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ad598c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad0137670 con 0x7f0ad01305c0 2024-09-21T03:28:32.545 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ad598c640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad01374f0 con 0x7f0ad012fc70 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ad598c640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f0ad01377f0 con 0x7f0ad0072d70 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ace7fc640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0ad012fc70 0x7f0ad009b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0aceffd640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0ad0072d70 0x7f0ad009b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0acf7fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad01305c0 0x7f0ad009bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ace7fc640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0ad012fc70 0x7f0ad009b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:55750/0 (socket says 172.21.15.27:55750) 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ace7fc640 1 -- 172.21.15.27:0/459113751 learned_addr learned my addr 172.21.15.27:0/459113751 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ace7fc640 1 -- 172.21.15.27:0/459113751 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0ad0072d70 msgr2=0x7f0ad009b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ace7fc640 1 --2- 172.21.15.27:0/459113751 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0ad0072d70 0x7f0ad009b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ace7fc640 1 -- 172.21.15.27:0/459113751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad01305c0 msgr2=0x7f0ad009bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ace7fc640 1 --2- 172.21.15.27:0/459113751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad01305c0 0x7f0ad009bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ace7fc640 1 -- 172.21.15.27:0/459113751 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad012de80 con 0x7f0ad012fc70 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0acf7fe640 1 --2- 172.21.15.27:0/459113751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad01305c0 0x7f0ad009bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:32.546 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ace7fc640 1 --2- 172.21.15.27:0/459113751 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0ad012fc70 0x7f0ad009b9c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800e9c0 tx=0x7f0ab800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:32.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ac7fff640 1 -- 172.21.15.27:0/459113751 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab800cd80 con 0x7f0ad012fc70 2024-09-21T03:28:32.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ac7fff640 1 -- 172.21.15.27:0/459113751 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0ab8004020 con 0x7f0ad012fc70 2024-09-21T03:28:32.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ad598c640 1 -- 172.21.15.27:0/459113751 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad012d100 con 0x7f0ad012fc70 2024-09-21T03:28:32.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ac7fff640 1 -- 172.21.15.27:0/459113751 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8004450 con 0x7f0ad012fc70 2024-09-21T03:28:32.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.542+0000 7f0ad598c640 1 -- 172.21.15.27:0/459113751 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad012e140 con 0x7f0ad012fc70 2024-09-21T03:28:32.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.544+0000 7f0ac7fff640 1 -- 172.21.15.27:0/459113751 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 8) ==== 63442+0+0 (secure 0 0 0) 0x7f0ab80035f0 con 0x7f0ad012fc70 2024-09-21T03:28:32.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.544+0000 7f0ad598c640 1 -- 172.21.15.27:0/459113751 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad00a08c0 con 0x7f0ad012fc70 2024-09-21T03:28:32.547 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.544+0000 7f0ac7fff640 1 -- 172.21.15.27:0/459113751 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 3027+0+0 (secure 0 0 0) 0x7f0ab8069850 con 0x7f0ad012fc70 2024-09-21T03:28:32.550 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.548+0000 7f0ac7fff640 1 -- 172.21.15.27:0/459113751 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f0ad00a08c0 con 0x7f0ad012fc70 2024-09-21T03:28:32.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.682+0000 7f0ac7fff640 1 -- 172.21.15.27:0/459113751 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f0ab80684b0 con 0x7f0ad012fc70 2024-09-21T03:28:32.723 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.721+0000 7f0ad598c640 1 -- 172.21.15.27:0/459113751 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd down", "ids": ["0"]} v 0) -- 0x7f0ad0130080 con 0x7f0ad012fc70 2024-09-21T03:28:32.723 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-21T03:28:32.721+0000 7f79be944640 -1 mon.a@0(leader).osd e22 definitely_dead 0 2024-09-21T03:28:32.724 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.722+0000 7f0ac7fff640 1 -- 172.21.15.27:0/459113751 <== mon.1 v2:172.21.15.27:3301/0 8 ==== mon_command_ack([{"prefix": "osd down", "ids": ["0"]}]=0 osd.0 is already down. v22) ==== 93+0+0 (secure 0 0 0) 0x7f0ab8004430 con 0x7f0ad012fc70 2024-09-21T03:28:32.724 INFO:teuthology.orchestra.run.smithi027.stderr:osd.0 is already down. 2024-09-21T03:28:32.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.723+0000 7f0ad598c640 1 -- 172.21.15.27:0/459113751 >> v1:172.21.15.27:6800/2405211194 conn(0x7f0ab0051420 legacy=0x7f0ab0053930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.723+0000 7f0ad598c640 1 -- 172.21.15.27:0/459113751 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0ad012fc70 msgr2=0x7f0ad009b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.723+0000 7f0ad598c640 1 --2- 172.21.15.27:0/459113751 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0ad012fc70 0x7f0ad009b9c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800e9c0 tx=0x7f0ab800ee80 comp rx=0 tx=0).stop 2024-09-21T03:28:32.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.724+0000 7f0ad598c640 1 -- 172.21.15.27:0/459113751 shutdown_connections 2024-09-21T03:28:32.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.724+0000 7f0ad598c640 1 --2- 172.21.15.27:0/459113751 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0ad01305c0 0x7f0ad009bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.724+0000 7f0ad598c640 1 --2- 172.21.15.27:0/459113751 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0ad012fc70 0x7f0ad009b9c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.726 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.724+0000 7f0ad598c640 1 --2- 172.21.15.27:0/459113751 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0ad0072d70 0x7f0ad009b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.724+0000 7f0ad598c640 1 -- 172.21.15.27:0/459113751 >> 172.21.15.27:0/459113751 conn(0x7f0ad00a1f40 msgr2=0x7f0ad01356d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.724+0000 7f0ad598c640 1 -- 172.21.15.27:0/459113751 shutdown_connections 2024-09-21T03:28:32.727 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.724+0000 7f0ad598c640 1 -- 172.21.15.27:0/459113751 wait complete. 2024-09-21T03:28:32.737 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T03:28:32.737 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-21T03:28:32.739 INFO:tasks.ceph.osd.0:Started 2024-09-21T03:28:32.739 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-21T03:28:32.740 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:28:32.775 INFO:tasks.ceph.osd.1:Stopped 2024-09-21T03:28:32.775 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-21T03:28:32.930 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.928+0000 7f36f458c640 1 -- 172.21.15.27:0/2171388335 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f36ec12ca00 msgr2=0x7f36ec12f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.928+0000 7f36f458c640 1 --2- 172.21.15.27:0/2171388335 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f36ec12ca00 0x7f36ec12f5f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f36dc009860 tx=0x7f36dc02a460 comp rx=0 tx=0).stop 2024-09-21T03:28:32.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.928+0000 7f36f458c640 1 -- 172.21.15.27:0/2171388335 shutdown_connections 2024-09-21T03:28:32.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.928+0000 7f36f458c640 1 --2- 172.21.15.27:0/2171388335 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f36ec12ca00 0x7f36ec12f5f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.928+0000 7f36f458c640 1 --2- 172.21.15.27:0/2171388335 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36ec0718e0 0x7f36ec12c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.928+0000 7f36f458c640 1 -- 172.21.15.27:0/2171388335 >> 172.21.15.27:0/2171388335 conn(0x7f36ec0a1d70 msgr2=0x7f36ec0a2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:32.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.928+0000 7f36f458c640 1 -- 172.21.15.27:0/2171388335 shutdown_connections 2024-09-21T03:28:32.931 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.928+0000 7f36f458c640 1 -- 172.21.15.27:0/2171388335 wait complete. 2024-09-21T03:28:32.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.929+0000 7f36f458c640 1 Processor -- start 2024-09-21T03:28:32.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.929+0000 7f36f458c640 1 -- start start 2024-09-21T03:28:32.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f458c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36ec0718e0 0x7f36ec131ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:32.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f458c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec12ca00 0x7f36ec1321e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:32.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f458c640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f36ec12fb30 0x7f36ec132720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:32.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f458c640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f36ec138730 con 0x7f36ec12ca00 2024-09-21T03:28:32.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f458c640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f36ec1385b0 con 0x7f36ec12fb30 2024-09-21T03:28:32.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f458c640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f36ec1388b0 con 0x7f36ec0718e0 2024-09-21T03:28:32.933 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f2328640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36ec0718e0 0x7f36ec131ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f2b29640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f36ec12fb30 0x7f36ec132720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f1b27640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec12ca00 0x7f36ec1321e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f2328640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36ec0718e0 0x7f36ec131ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:43380/0 (socket says 172.21.15.27:43380) 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f1b27640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec12ca00 0x7f36ec1321e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44434/0 (socket says 172.21.15.27:44434) 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f2b29640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f36ec12fb30 0x7f36ec132720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:55772/0 (socket says 172.21.15.27:55772) 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f2328640 1 -- 172.21.15.27:0/2076619052 learned_addr learned my addr 172.21.15.27:0/2076619052 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f2328640 1 -- 172.21.15.27:0/2076619052 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f36ec12fb30 msgr2=0x7f36ec132720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f2328640 1 --2- 172.21.15.27:0/2076619052 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f36ec12fb30 0x7f36ec132720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f2328640 1 -- 172.21.15.27:0/2076619052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec12ca00 msgr2=0x7f36ec1321e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f2328640 1 --2- 172.21.15.27:0/2076619052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec12ca00 0x7f36ec1321e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f2328640 1 -- 172.21.15.27:0/2076619052 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36ec134cf0 con 0x7f36ec0718e0 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f1b27640 1 --2- 172.21.15.27:0/2076619052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec12ca00 0x7f36ec1321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:32.934 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.930+0000 7f36f2328640 1 --2- 172.21.15.27:0/2076619052 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36ec0718e0 0x7f36ec131ca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f36e800e9f0 tx=0x7f36e800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.931+0000 7f36e2ffd640 1 -- 172.21.15.27:0/2076619052 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e800cd80 con 0x7f36ec0718e0 2024-09-21T03:28:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.931+0000 7f36e2ffd640 1 -- 172.21.15.27:0/2076619052 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f36e8004020 con 0x7f36ec0718e0 2024-09-21T03:28:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.931+0000 7f36e2ffd640 1 -- 172.21.15.27:0/2076619052 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e8004470 con 0x7f36ec0718e0 2024-09-21T03:28:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.931+0000 7f36f458c640 1 -- 172.21.15.27:0/2076619052 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36ec1346a0 con 0x7f36ec0718e0 2024-09-21T03:28:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.931+0000 7f36f458c640 1 -- 172.21.15.27:0/2076619052 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f36ec1cbd90 con 0x7f36ec0718e0 2024-09-21T03:28:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.932+0000 7f36f458c640 1 -- 172.21.15.27:0/2076619052 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36ec133f30 con 0x7f36ec0718e0 2024-09-21T03:28:32.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.932+0000 7f36e2ffd640 1 -- 172.21.15.27:0/2076619052 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f36e80035f0 con 0x7f36ec0718e0 2024-09-21T03:28:32.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.932+0000 7f36e2ffd640 1 -- 172.21.15.27:0/2076619052 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(22..22 src has 1..22) ==== 3027+0+0 (secure 0 0 0) 0x7f36e8069a50 con 0x7f36ec0718e0 2024-09-21T03:28:32.938 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:32.936+0000 7f36e2ffd640 1 -- 172.21.15.27:0/2076619052 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f36e8009490 con 0x7f36ec0718e0 2024-09-21T03:28:33.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.106+0000 7f36f458c640 1 -- 172.21.15.27:0/2076619052 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd down", "ids": ["1"]} v 0) -- 0x7f36ec12c4c0 con 0x7f36ec0718e0 2024-09-21T03:28:33.110 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-21T03:28:33.106+0000 7f79be944640 -1 mon.a@0(leader).osd e22 definitely_dead 0 2024-09-21T03:28:33.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.683+0000 7f36e2ffd640 1 -- 172.21.15.27:0/2076619052 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd down", "ids": ["1"]}]=0 marked down osd.1. v23) ==== 89+0+0 (secure 0 0 0) 0x7f36e8068b00 con 0x7f36ec0718e0 2024-09-21T03:28:33.686 INFO:teuthology.orchestra.run.smithi027.stderr:marked down osd.1. 2024-09-21T03:28:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.686+0000 7f36f458c640 1 -- 172.21.15.27:0/2076619052 >> v1:172.21.15.27:6800/2405211194 conn(0x7f36d00514b0 legacy=0x7f36d00539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.686+0000 7f36f458c640 1 -- 172.21.15.27:0/2076619052 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36ec0718e0 msgr2=0x7f36ec131ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.686+0000 7f36f458c640 1 --2- 172.21.15.27:0/2076619052 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36ec0718e0 0x7f36ec131ca0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f36e800e9f0 tx=0x7f36e800eeb0 comp rx=0 tx=0).stop 2024-09-21T03:28:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.686+0000 7f36f458c640 1 -- 172.21.15.27:0/2076619052 shutdown_connections 2024-09-21T03:28:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.686+0000 7f36f458c640 1 --2- 172.21.15.27:0/2076619052 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f36ec12fb30 0x7f36ec132720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.686+0000 7f36f458c640 1 --2- 172.21.15.27:0/2076619052 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f36ec12ca00 0x7f36ec1321e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.686+0000 7f36f458c640 1 --2- 172.21.15.27:0/2076619052 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36ec0718e0 0x7f36ec131ca0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:33.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.686+0000 7f36f458c640 1 -- 172.21.15.27:0/2076619052 >> 172.21.15.27:0/2076619052 conn(0x7f36ec0a1d70 msgr2=0x7f36ec126c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.686+0000 7f36f458c640 1 -- 172.21.15.27:0/2076619052 shutdown_connections 2024-09-21T03:28:33.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.687+0000 7f36f458c640 1 -- 172.21.15.27:0/2076619052 wait complete. 2024-09-21T03:28:33.701 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-21T03:28:33.701 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-21T03:28:33.703 INFO:tasks.ceph.osd.1:Started 2024-09-21T03:28:33.703 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-21T03:28:33.703 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:28:33.739 INFO:tasks.ceph.osd.2:Stopped 2024-09-21T03:28:33.740 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2 2024-09-21T03:28:33.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.893+0000 7f1996899640 1 -- 172.21.15.27:0/2674481327 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f199012a840 msgr2=0x7f199012d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:33.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.893+0000 7f1996899640 1 --2- 172.21.15.27:0/2674481327 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f199012a840 0x7f199012d4c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1978009830 tx=0x7f197802a3f0 comp rx=0 tx=0).stop 2024-09-21T03:28:33.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.893+0000 7f1996899640 1 -- 172.21.15.27:0/2674481327 shutdown_connections 2024-09-21T03:28:33.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.893+0000 7f1996899640 1 --2- 172.21.15.27:0/2674481327 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f199012a840 0x7f199012d4c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:33.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.893+0000 7f1996899640 1 --2- 172.21.15.27:0/2674481327 >> v2:172.21.15.27:3300/0 conn(0x7f1990073d60 0x7f199012a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:33.897 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.893+0000 7f1996899640 1 -- 172.21.15.27:0/2674481327 >> 172.21.15.27:0/2674481327 conn(0x7f199009fe40 msgr2=0x7f19900a0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:33.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.893+0000 7f1996899640 1 -- 172.21.15.27:0/2674481327 shutdown_connections 2024-09-21T03:28:33.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.893+0000 7f1996899640 1 -- 172.21.15.27:0/2674481327 wait complete. 2024-09-21T03:28:33.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.894+0000 7f1996899640 1 Processor -- start 2024-09-21T03:28:33.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.894+0000 7f1996899640 1 -- start start 2024-09-21T03:28:33.901 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1996899640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1990073d60 0x7f19901346c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1996899640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199012a840 0x7f1990134c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1996899640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1990135140 0x7f19901c2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1996899640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f19901363a0 con 0x7f199012a840 2024-09-21T03:28:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1996899640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f1990136220 con 0x7f1990135140 2024-09-21T03:28:33.902 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1996899640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f1990136520 con 0x7f1990073d60 2024-09-21T03:28:33.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1987fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199012a840 0x7f1990134c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:33.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1994e36640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1990135140 0x7f19901c2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:33.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f198ffff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1990073d60 0x7f19901346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:33.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1987fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199012a840 0x7f1990134c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44462/0 (socket says 172.21.15.27:44462) 2024-09-21T03:28:33.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1987fff640 1 -- 172.21.15.27:0/3179943267 learned_addr learned my addr 172.21.15.27:0/3179943267 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:33.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1987fff640 1 -- 172.21.15.27:0/3179943267 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1990073d60 msgr2=0x7f19901346c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:33.903 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1987fff640 1 --2- 172.21.15.27:0/3179943267 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1990073d60 0x7f19901346c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1987fff640 1 -- 172.21.15.27:0/3179943267 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1990135140 msgr2=0x7f19901c2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1987fff640 1 --2- 172.21.15.27:0/3179943267 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1990135140 0x7f19901c2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1987fff640 1 -- 172.21.15.27:0/3179943267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19901336a0 con 0x7f199012a840 2024-09-21T03:28:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.895+0000 7f1994e36640 1 --2- 172.21.15.27:0/3179943267 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1990135140 0x7f19901c2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.896+0000 7f1987fff640 1 --2- 172.21.15.27:0/3179943267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199012a840 0x7f1990134c00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1978009950 tx=0x7f1978032c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.896+0000 7f198d7fa640 1 -- 172.21.15.27:0/3179943267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1978040070 con 0x7f199012a840 2024-09-21T03:28:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.896+0000 7f198d7fa640 1 -- 172.21.15.27:0/3179943267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1978005310 con 0x7f199012a840 2024-09-21T03:28:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.896+0000 7f198d7fa640 1 -- 172.21.15.27:0/3179943267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1978005c90 con 0x7f199012a840 2024-09-21T03:28:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.896+0000 7f1996899640 1 -- 172.21.15.27:0/3179943267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1990132ed0 con 0x7f199012a840 2024-09-21T03:28:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.896+0000 7f1996899640 1 -- 172.21.15.27:0/3179943267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19901c33a0 con 0x7f199012a840 2024-09-21T03:28:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.897+0000 7f1996899640 1 -- 172.21.15.27:0/3179943267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f199012ac50 con 0x7f199012a840 2024-09-21T03:28:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.897+0000 7f198d7fa640 1 -- 172.21.15.27:0/3179943267 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f1978039040 con 0x7f199012a840 2024-09-21T03:28:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.898+0000 7f198d7fa640 1 -- 172.21.15.27:0/3179943267 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 3027+0+0 (secure 0 0 0) 0x7f197808a0c0 con 0x7f199012a840 2024-09-21T03:28:33.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:33.901+0000 7f198d7fa640 1 -- 172.21.15.27:0/3179943267 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f1978089170 con 0x7f199012a840 2024-09-21T03:28:34.073 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.070+0000 7f1996899640 1 -- 172.21.15.27:0/3179943267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd down", "ids": ["2"]} v 0) -- 0x7f1990134000 con 0x7f199012a840 2024-09-21T03:28:34.073 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-21T03:28:34.071+0000 7f79be944640 -1 mon.a@0(leader).osd e23 definitely_dead 0 2024-09-21T03:28:34.179 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-21T03:28:34.176+0000 7f4173018640 -1 Falling back to public interface 2024-09-21T03:28:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.685+0000 7f198d7fa640 1 -- 172.21.15.27:0/3179943267 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd down", "ids": ["2"]}]=0 marked down osd.2. v24) ==== 89+0+0 (secure 0 0 0) 0x7f197803fd90 con 0x7f199012a840 2024-09-21T03:28:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:marked down osd.2. 2024-09-21T03:28:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.688+0000 7f1996899640 1 -- 172.21.15.27:0/3179943267 >> v1:172.21.15.27:6800/2405211194 conn(0x7f19700514b0 legacy=0x7f19700539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.688+0000 7f1996899640 1 -- 172.21.15.27:0/3179943267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199012a840 msgr2=0x7f1990134c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.688+0000 7f1996899640 1 --2- 172.21.15.27:0/3179943267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199012a840 0x7f1990134c00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1978009950 tx=0x7f1978032c10 comp rx=0 tx=0).stop 2024-09-21T03:28:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.689+0000 7f1996899640 1 -- 172.21.15.27:0/3179943267 shutdown_connections 2024-09-21T03:28:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.689+0000 7f1996899640 1 --2- 172.21.15.27:0/3179943267 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1990135140 0x7f19901c2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.689+0000 7f1996899640 1 --2- 172.21.15.27:0/3179943267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f199012a840 0x7f1990134c00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.689+0000 7f1996899640 1 --2- 172.21.15.27:0/3179943267 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1990073d60 0x7f19901346c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.689+0000 7f1996899640 1 -- 172.21.15.27:0/3179943267 >> 172.21.15.27:0/3179943267 conn(0x7f199009fe40 msgr2=0x7f199012d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.690+0000 7f1996899640 1 -- 172.21.15.27:0/3179943267 shutdown_connections 2024-09-21T03:28:34.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.690+0000 7f1996899640 1 -- 172.21.15.27:0/3179943267 wait complete. 2024-09-21T03:28:34.705 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T03:28:34.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-21T03:28:34.708 INFO:tasks.ceph.osd.2:Started 2024-09-21T03:28:34.708 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-21T03:28:34.708 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:28:34.748 INFO:tasks.ceph.osd.3:Stopped 2024-09-21T03:28:34.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2024-09-21T03:28:34.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.902+0000 7f121c738640 1 -- 172.21.15.27:0/2469577613 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f121412a8d0 msgr2=0x7f121412d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:34.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.902+0000 7f121c738640 1 --2- 172.21.15.27:0/2469577613 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f121412a8d0 0x7f121412d5e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1210009be0 tx=0x7f121002a800 comp rx=0 tx=0).stop 2024-09-21T03:28:34.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.903+0000 7f121c738640 1 -- 172.21.15.27:0/2469577613 shutdown_connections 2024-09-21T03:28:34.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.903+0000 7f121c738640 1 --2- 172.21.15.27:0/2469577613 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f121412a8d0 0x7f121412d5e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:34.905 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.903+0000 7f121c738640 1 --2- 172.21.15.27:0/2469577613 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1214073c10 0x7f121412a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:34.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.903+0000 7f121c738640 1 -- 172.21.15.27:0/2469577613 >> 172.21.15.27:0/2469577613 conn(0x7f121409fa20 msgr2=0x7f121409fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:34.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.903+0000 7f121c738640 1 -- 172.21.15.27:0/2469577613 shutdown_connections 2024-09-21T03:28:34.906 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.903+0000 7f121c738640 1 -- 172.21.15.27:0/2469577613 wait complete. 2024-09-21T03:28:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121c738640 1 Processor -- start 2024-09-21T03:28:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121c738640 1 -- start start 2024-09-21T03:28:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121c738640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1214073c10 0x7f121406faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121c738640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f121412a8d0 0x7f1214070030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121c738640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f121412ace0 0x7f1214070570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121c738640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1214136670 con 0x7f121412ace0 2024-09-21T03:28:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121c738640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f12141364f0 con 0x7f1214073c10 2024-09-21T03:28:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121c738640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f12141367f0 con 0x7f121412a8d0 2024-09-21T03:28:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121a4d4640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1214073c10 0x7f121406faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f1219cd3640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f121412a8d0 0x7f1214070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:34.908 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121a4d4640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1214073c10 0x7f121406faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:55834/0 (socket says 172.21.15.27:55834) 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121a4d4640 1 -- 172.21.15.27:0/3103748314 learned_addr learned my addr 172.21.15.27:0/3103748314 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f1219cd3640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f121412a8d0 0x7f1214070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:43434/0 (socket says 172.21.15.27:43434) 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.904+0000 7f121acd5640 1 --2- 172.21.15.27:0/3103748314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f121412ace0 0x7f1214070570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.905+0000 7f121a4d4640 1 -- 172.21.15.27:0/3103748314 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f121412a8d0 msgr2=0x7f1214070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.905+0000 7f121a4d4640 1 --2- 172.21.15.27:0/3103748314 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f121412a8d0 0x7f1214070030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.905+0000 7f121a4d4640 1 -- 172.21.15.27:0/3103748314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f121412ace0 msgr2=0x7f1214070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.905+0000 7f121a4d4640 1 --2- 172.21.15.27:0/3103748314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f121412ace0 0x7f1214070570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.905+0000 7f121a4d4640 1 -- 172.21.15.27:0/3103748314 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1214134af0 con 0x7f1214073c10 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.905+0000 7f121a4d4640 1 --2- 172.21.15.27:0/3103748314 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1214073c10 0x7f121406faf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f120800e9f0 tx=0x7f120800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.905+0000 7f1206ffd640 1 -- 172.21.15.27:0/3103748314 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f120800cd80 con 0x7f1214073c10 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.905+0000 7f1206ffd640 1 -- 172.21.15.27:0/3103748314 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1208004020 con 0x7f1214073c10 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.905+0000 7f1206ffd640 1 -- 172.21.15.27:0/3103748314 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1208004470 con 0x7f1214073c10 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.905+0000 7f121c738640 1 -- 172.21.15.27:0/3103748314 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12141c5ae0 con 0x7f1214073c10 2024-09-21T03:28:34.909 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.905+0000 7f121c738640 1 -- 172.21.15.27:0/3103748314 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12141c6080 con 0x7f1214073c10 2024-09-21T03:28:34.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.906+0000 7f1206ffd640 1 -- 172.21.15.27:0/3103748314 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f12080035f0 con 0x7f1214073c10 2024-09-21T03:28:34.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.906+0000 7f121c738640 1 -- 172.21.15.27:0/3103748314 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f121408c370 con 0x7f1214073c10 2024-09-21T03:28:34.910 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.907+0000 7f1206ffd640 1 -- 172.21.15.27:0/3103748314 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 3027+0+0 (secure 0 0 0) 0x7f1208069890 con 0x7f1214073c10 2024-09-21T03:28:34.913 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:34.911+0000 7f1206ffd640 1 -- 172.21.15.27:0/3103748314 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f121408c370 con 0x7f1214073c10 2024-09-21T03:28:35.080 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.078+0000 7f121c738640 1 -- 172.21.15.27:0/3103748314 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd down", "ids": ["3"]} v 0) -- 0x7f121412a390 con 0x7f1214073c10 2024-09-21T03:28:35.081 INFO:tasks.ceph.mon.a.smithi027.stderr:2024-09-21T03:28:35.078+0000 7f79be944640 -1 mon.a@0(leader).osd e24 definitely_dead 0 2024-09-21T03:28:35.119 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-09-21T03:28:35.117+0000 7f9464616640 -1 Falling back to public interface 2024-09-21T03:28:35.693 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.691+0000 7f1206ffd640 1 -- 172.21.15.27:0/3103748314 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd down", "ids": ["3"]}]=0 marked down osd.3. v25) ==== 89+0+0 (secure 0 0 0) 0x7f1208009490 con 0x7f1214073c10 2024-09-21T03:28:35.693 INFO:teuthology.orchestra.run.smithi027.stderr:marked down osd.3. 2024-09-21T03:28:35.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.693+0000 7f121c738640 1 -- 172.21.15.27:0/3103748314 >> v1:172.21.15.27:6800/2405211194 conn(0x7f11f80514b0 legacy=0x7f11f80539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:35.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.693+0000 7f121c738640 1 -- 172.21.15.27:0/3103748314 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1214073c10 msgr2=0x7f121406faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:35.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.693+0000 7f121c738640 1 --2- 172.21.15.27:0/3103748314 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1214073c10 0x7f121406faf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f120800e9f0 tx=0x7f120800eeb0 comp rx=0 tx=0).stop 2024-09-21T03:28:35.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.693+0000 7f121c738640 1 -- 172.21.15.27:0/3103748314 shutdown_connections 2024-09-21T03:28:35.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.693+0000 7f121c738640 1 --2- 172.21.15.27:0/3103748314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f121412ace0 0x7f1214070570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:35.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.693+0000 7f121c738640 1 --2- 172.21.15.27:0/3103748314 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f121412a8d0 0x7f1214070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:35.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.693+0000 7f121c738640 1 --2- 172.21.15.27:0/3103748314 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1214073c10 0x7f121406faf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:35.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.693+0000 7f121c738640 1 -- 172.21.15.27:0/3103748314 >> 172.21.15.27:0/3103748314 conn(0x7f121409fa20 msgr2=0x7f12140994b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:35.696 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.694+0000 7f121c738640 1 -- 172.21.15.27:0/3103748314 shutdown_connections 2024-09-21T03:28:35.697 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.694+0000 7f121c738640 1 -- 172.21.15.27:0/3103748314 wait complete. 2024-09-21T03:28:35.710 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-21T03:28:35.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-21T03:28:35.712 INFO:tasks.ceph.osd.3:Started 2024-09-21T03:28:35.713 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T03:28:35.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:28:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.829+0000 7f266cbbc640 1 -- 172.21.15.27:0/1519575848 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2668073c10 msgr2=0x7f266812a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.829+0000 7f266cbbc640 1 --2- 172.21.15.27:0/1519575848 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2668073c10 0x7f266812a390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2658005fd0 tx=0x7f265802a540 comp rx=0 tx=0).stop 2024-09-21T03:28:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.829+0000 7f266cbbc640 1 -- 172.21.15.27:0/1519575848 shutdown_connections 2024-09-21T03:28:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.829+0000 7f266cbbc640 1 --2- 172.21.15.27:0/1519575848 >> v2:172.21.15.27:3300/0 conn(0x7f266812ace0 0x7f266812dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:35.832 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.829+0000 7f266cbbc640 1 --2- 172.21.15.27:0/1519575848 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2668073c10 0x7f266812a390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.829+0000 7f266cbbc640 1 -- 172.21.15.27:0/1519575848 >> 172.21.15.27:0/1519575848 conn(0x7f266809fa20 msgr2=0x7f266809fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.830+0000 7f266cbbc640 1 -- 172.21.15.27:0/1519575848 shutdown_connections 2024-09-21T03:28:35.833 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.830+0000 7f266cbbc640 1 -- 172.21.15.27:0/1519575848 wait complete. 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.831+0000 7f266cbbc640 1 Processor -- start 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.831+0000 7f266cbbc640 1 -- start start 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.831+0000 7f266cbbc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668073c10 0x7f2668133c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.831+0000 7f266cbbc640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f266812a8d0 0x7f26681341a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.831+0000 7f266cbbc640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f266812ace0 0x7f26681346e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.831+0000 7f266cbbc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f26681365e0 con 0x7f2668073c10 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.831+0000 7f266cbbc640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f2668136460 con 0x7f266812a8d0 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.831+0000 7f266cbbc640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f2668136760 con 0x7f266812ace0 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.831+0000 7f266659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668073c10 0x7f2668133c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.831+0000 7f2665d9b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f266812a8d0 0x7f26681341a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.831+0000 7f2666d9d640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f266812ace0 0x7f26681346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:35.835 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f266659c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668073c10 0x7f2668133c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44488/0 (socket says 172.21.15.27:44488) 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f266659c640 1 -- 172.21.15.27:0/917797883 learned_addr learned my addr 172.21.15.27:0/917797883 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f266659c640 1 -- 172.21.15.27:0/917797883 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f266812ace0 msgr2=0x7f26681346e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f266659c640 1 --2- 172.21.15.27:0/917797883 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f266812ace0 0x7f26681346e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f266659c640 1 -- 172.21.15.27:0/917797883 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f266812a8d0 msgr2=0x7f26681341a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f266659c640 1 --2- 172.21.15.27:0/917797883 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f266812a8d0 0x7f26681341a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f266659c640 1 -- 172.21.15.27:0/917797883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2668132840 con 0x7f2668073c10 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f2666d9d640 1 --2- 172.21.15.27:0/917797883 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f266812ace0 0x7f26681346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f266659c640 1 --2- 172.21.15.27:0/917797883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668073c10 0x7f2668133c60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2658006010 tx=0x7f2658005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f2656ffd640 1 -- 172.21.15.27:0/917797883 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f265803e030 con 0x7f2668073c10 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f2656ffd640 1 -- 172.21.15.27:0/917797883 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2658003c30 con 0x7f2668073c10 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f266cbbc640 1 -- 172.21.15.27:0/917797883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2668134c20 con 0x7f2668073c10 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f2656ffd640 1 -- 172.21.15.27:0/917797883 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658035040 con 0x7f2668073c10 2024-09-21T03:28:35.836 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.832+0000 7f266cbbc640 1 -- 172.21.15.27:0/917797883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26681c9a60 con 0x7f2668073c10 2024-09-21T03:28:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.833+0000 7f2656ffd640 1 -- 172.21.15.27:0/917797883 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f2658041030 con 0x7f2668073c10 2024-09-21T03:28:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.834+0000 7f266cbbc640 1 -- 172.21.15.27:0/917797883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f266812a390 con 0x7f2668073c10 2024-09-21T03:28:35.837 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.834+0000 7f2656ffd640 1 -- 172.21.15.27:0/917797883 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 3027+0+0 (secure 0 0 0) 0x7f2658089550 con 0x7f2668073c10 2024-09-21T03:28:35.840 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.838+0000 7f2656ffd640 1 -- 172.21.15.27:0/917797883 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f2658088600 con 0x7f2668073c10 2024-09-21T03:28:35.904 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-09-21T03:28:35.902+0000 7f3bf5a0b640 -1 Falling back to public interface 2024-09-21T03:28:35.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.997+0000 7f266cbbc640 1 -- 172.21.15.27:0/917797883 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2668069f90 con 0x7f2668073c10 2024-09-21T03:28:36.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.998+0000 7f2656ffd640 1 -- 172.21.15.27:0/917797883 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+8650 (secure 0 0 0) 0x7f2658088140 con 0x7f2668073c10 2024-09-21T03:28:36.000 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:36.000 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":25,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:28:35.686744+0000","last_up_change":"2024-09-21T03:28:35.686744+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"squid","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T03:27:11.408521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6809","nonce":831015510}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6810","nonce":831015510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6812","nonce":831015510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6811","nonce":831015510}]},"public_addr":"172.21.15.27:6809/831015510","cluster_addr":"172.21.15.27:6810/831015510","heartbeat_back_addr":"172.21.15.27:6812/831015510","heartbeat_front_addr":"172.21.15.27:6811/831015510","state":["exists"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":22,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":1590434734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":1590434734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":1590434734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":1590434734}]},"public_addr":"172.21.15.27:6805/1590434734","cluster_addr":"172.21.15.27:6806/1590434734","heartbeat_back_addr":"172.21.15.27:6808/1590434734","heartbeat_front_addr":"172.21.15.27:6807/1590434734","state":["exists"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":3010908863}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":3010908863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":3010908863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":3010908863}]},"public_addr":"172.21.15.27:6801/3010908863","cluster_addr":"172.21.15.27:6802/3010908863","heartbeat_back_addr":"172.21.15.27:6804/3010908863","heartbeat_front_addr":"172.21.15.27:6803/3010908863","state":["exists"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":24,"down_at":25,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1661572621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1661572621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1661572621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1661572621}]},"public_addr":"172.21.15.27:6813/1661572621","cluster_addr":"172.21.15.27:6814/1661572621","heartbeat_back_addr":"172.21.15.27:6816/1661572621","heartbeat_front_addr":"172.21.15.27:6815/1661572621","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":22},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":23},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":24},{"osd":3,"down_stamp":"2024-09-21T03:28:35.686744+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1436650201":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3459873364":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3140689679":"2024-09-22T03:28:28.273799+0000","172.21.15.27:6800/1098903051":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/1012963840":"2024-09-22T03:28:28.273799+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:28:36.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.999+0000 7f266cbbc640 1 -- 172.21.15.27:0/917797883 >> v1:172.21.15.27:6800/2405211194 conn(0x7f26480514b0 legacy=0x7f26480539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:36.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.999+0000 7f266cbbc640 1 -- 172.21.15.27:0/917797883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668073c10 msgr2=0x7f2668133c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:36.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.999+0000 7f266cbbc640 1 --2- 172.21.15.27:0/917797883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668073c10 0x7f2668133c60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2658006010 tx=0x7f2658005e80 comp rx=0 tx=0).stop 2024-09-21T03:28:36.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.999+0000 7f266cbbc640 1 -- 172.21.15.27:0/917797883 shutdown_connections 2024-09-21T03:28:36.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.999+0000 7f266cbbc640 1 --2- 172.21.15.27:0/917797883 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f266812ace0 0x7f26681346e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:36.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.999+0000 7f266cbbc640 1 --2- 172.21.15.27:0/917797883 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f266812a8d0 0x7f26681341a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:36.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.999+0000 7f266cbbc640 1 --2- 172.21.15.27:0/917797883 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2668073c10 0x7f2668133c60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:36.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.999+0000 7f266cbbc640 1 -- 172.21.15.27:0/917797883 >> 172.21.15.27:0/917797883 conn(0x7f266809fa20 msgr2=0x7f2668099540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:36.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.999+0000 7f266cbbc640 1 -- 172.21.15.27:0/917797883 shutdown_connections 2024-09-21T03:28:36.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:35.999+0000 7f266cbbc640 1 -- 172.21.15.27:0/917797883 wait complete. 2024-09-21T03:28:37.169 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-09-21T03:28:37.168+0000 7f32c2e14640 -1 Falling back to public interface 2024-09-21T03:28:37.754 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-21T03:28:37.753+0000 7f4173018640 -1 osd.0 21 log_to_monitors true 2024-09-21T03:28:37.825 INFO:tasks.ceph.osd.0.smithi027.stderr:2024-09-21T03:28:37.823+0000 7f4166694640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T03:28:38.746 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-09-21T03:28:38.744+0000 7f9464616640 -1 osd.1 22 log_to_monitors true 2024-09-21T03:28:38.828 INFO:tasks.ceph.osd.1.smithi027.stderr:2024-09-21T03:28:38.827+0000 7f9457e48640 -1 osd.1 22 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T03:28:39.013 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:28:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.126+0000 7fb5c2072640 1 -- 172.21.15.27:0/4041425850 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb5bc130000 msgr2=0x7fb5bc12c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.126+0000 7fb5c2072640 1 --2- 172.21.15.27:0/4041425850 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb5bc130000 0x7fb5bc12c110 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0005fd0 tx=0x7fb5b002a4a0 comp rx=0 tx=0).stop 2024-09-21T03:28:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.126+0000 7fb5c2072640 1 -- 172.21.15.27:0/4041425850 shutdown_connections 2024-09-21T03:28:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.126+0000 7fb5c2072640 1 --2- 172.21.15.27:0/4041425850 >> v2:172.21.15.27:3300/0 conn(0x7fb5bc12c650 0x7fb5bc12ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.126+0000 7fb5c2072640 1 --2- 172.21.15.27:0/4041425850 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb5bc130000 0x7fb5bc12c110 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.126+0000 7fb5c2072640 1 --2- 172.21.15.27:0/4041425850 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb5bc071720 0x7fb5bc12fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.126+0000 7fb5c2072640 1 -- 172.21.15.27:0/4041425850 >> 172.21.15.27:0/4041425850 conn(0x7fb5bc0a1e10 msgr2=0x7fb5bc0a2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:39.128 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.126+0000 7fb5c2072640 1 -- 172.21.15.27:0/4041425850 shutdown_connections 2024-09-21T03:28:39.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.127+0000 7fb5c2072640 1 -- 172.21.15.27:0/4041425850 wait complete. 2024-09-21T03:28:39.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.128+0000 7fb5c2072640 1 Processor -- start 2024-09-21T03:28:39.129 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.128+0000 7fb5c2072640 1 -- start start 2024-09-21T03:28:39.130 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.128+0000 7fb5c2072640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb5bc071720 0x7fb5bc09b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.128+0000 7fb5c2072640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5bc12c650 0x7fb5bc09b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.128+0000 7fb5c2072640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb5bc130000 0x7fb5bc09bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.128+0000 7fb5c2072640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc137490 con 0x7fb5bc12c650 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.128+0000 7fb5c2072640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fb5bc137310 con 0x7fb5bc071720 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.128+0000 7fb5c2072640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fb5bc137610 con 0x7fb5bc130000 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5bb7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb5bc071720 0x7fb5bc09b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5baffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5bc12c650 0x7fb5bc09b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5bbfff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb5bc130000 0x7fb5bc09bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5baffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5bc12c650 0x7fb5bc09b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44516/0 (socket says 172.21.15.27:44516) 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5bb7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb5bc071720 0x7fb5bc09b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:55858/0 (socket says 172.21.15.27:55858) 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5baffd640 1 -- 172.21.15.27:0/916762646 learned_addr learned my addr 172.21.15.27:0/916762646 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:39.131 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5baffd640 1 -- 172.21.15.27:0/916762646 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb5bc130000 msgr2=0x7fb5bc09bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:39.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5baffd640 1 --2- 172.21.15.27:0/916762646 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb5bc130000 0x7fb5bc09bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:39.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5baffd640 1 -- 172.21.15.27:0/916762646 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb5bc071720 msgr2=0x7fb5bc09b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:39.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5baffd640 1 --2- 172.21.15.27:0/916762646 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb5bc071720 0x7fb5bc09b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:39.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5baffd640 1 -- 172.21.15.27:0/916762646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5bc1349b0 con 0x7fb5bc12c650 2024-09-21T03:28:39.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5baffd640 1 --2- 172.21.15.27:0/916762646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5bc12c650 0x7fb5bc09b9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0034040 tx=0x7fb5b0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:39.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5a3fff640 1 -- 172.21.15.27:0/916762646 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b003e030 con 0x7fb5bc12c650 2024-09-21T03:28:39.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5a3fff640 1 -- 172.21.15.27:0/916762646 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fb5b0003c30 con 0x7fb5bc12c650 2024-09-21T03:28:39.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5a3fff640 1 -- 172.21.15.27:0/916762646 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0035040 con 0x7fb5bc12c650 2024-09-21T03:28:39.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5c2072640 1 -- 172.21.15.27:0/916762646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5bc133c30 con 0x7fb5bc12c650 2024-09-21T03:28:39.132 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.129+0000 7fb5c2072640 1 -- 172.21.15.27:0/916762646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5bc134c70 con 0x7fb5bc12c650 2024-09-21T03:28:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.130+0000 7fb5c2072640 1 -- 172.21.15.27:0/916762646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5bc12ca60 con 0x7fb5bc12c650 2024-09-21T03:28:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.130+0000 7fb5a3fff640 1 -- 172.21.15.27:0/916762646 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7fb5b0041030 con 0x7fb5bc12c650 2024-09-21T03:28:39.133 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.131+0000 7fb5a3fff640 1 -- 172.21.15.27:0/916762646 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3027+0+0 (secure 0 0 0) 0x7fb5b0089640 con 0x7fb5bc12c650 2024-09-21T03:28:39.136 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.135+0000 7fb5a3fff640 1 -- 172.21.15.27:0/916762646 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fb5b00886f0 con 0x7fb5bc12c650 2024-09-21T03:28:39.296 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.295+0000 7fb5c2072640 1 -- 172.21.15.27:0/916762646 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5bc06dda0 con 0x7fb5bc12c650 2024-09-21T03:28:39.297 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.296+0000 7fb5a3fff640 1 -- 172.21.15.27:0/916762646 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+9099 (secure 0 0 0) 0x7fb5b0088230 con 0x7fb5bc12c650 2024-09-21T03:28:39.297 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:39.297 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:28:38.816537+0000","last_up_change":"2024-09-21T03:28:38.816537+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"squid","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.25,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T03:27:11.408521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":21,"up_from":27,"up_thru":0,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":1914045614}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":1914045614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":1914045614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":1914045614}]},"public_addr":"172.21.15.27:6801/1914045614","cluster_addr":"172.21.15.27:6802/1914045614","heartbeat_back_addr":"172.21.15.27:6804/1914045614","heartbeat_front_addr":"172.21.15.27:6803/1914045614","state":["exists","up"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":22,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":1590434734}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":1590434734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":1590434734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":1590434734}]},"public_addr":"172.21.15.27:6805/1590434734","cluster_addr":"172.21.15.27:6806/1590434734","heartbeat_back_addr":"172.21.15.27:6808/1590434734","heartbeat_front_addr":"172.21.15.27:6807/1590434734","state":["exists"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":3010908863}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":3010908863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":3010908863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":3010908863}]},"public_addr":"172.21.15.27:6801/3010908863","cluster_addr":"172.21.15.27:6802/3010908863","heartbeat_back_addr":"172.21.15.27:6804/3010908863","heartbeat_front_addr":"172.21.15.27:6803/3010908863","state":["exists"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":24,"down_at":25,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1661572621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1661572621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1661572621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1661572621}]},"public_addr":"172.21.15.27:6813/1661572621","cluster_addr":"172.21.15.27:6814/1661572621","heartbeat_back_addr":"172.21.15.27:6816/1661572621","heartbeat_front_addr":"172.21.15.27:6815/1661572621","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":22},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":23},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":24},{"osd":3,"down_stamp":"2024-09-21T03:28:35.686744+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1436650201":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3459873364":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3140689679":"2024-09-22T03:28:28.273799+0000","172.21.15.27:6800/1098903051":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/1012963840":"2024-09-22T03:28:28.273799+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:28:39.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.298+0000 7fb5c2072640 1 -- 172.21.15.27:0/916762646 >> v1:172.21.15.27:6800/2405211194 conn(0x7fb5a80514a0 legacy=0x7fb5a80539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:39.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.298+0000 7fb5c2072640 1 -- 172.21.15.27:0/916762646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5bc12c650 msgr2=0x7fb5bc09b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:39.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.298+0000 7fb5c2072640 1 --2- 172.21.15.27:0/916762646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5bc12c650 0x7fb5bc09b9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0034040 tx=0x7fb5b0005e80 comp rx=0 tx=0).stop 2024-09-21T03:28:39.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.298+0000 7fb5c2072640 1 -- 172.21.15.27:0/916762646 shutdown_connections 2024-09-21T03:28:39.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.298+0000 7fb5c2072640 1 --2- 172.21.15.27:0/916762646 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fb5bc130000 0x7fb5bc09bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:39.299 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.298+0000 7fb5c2072640 1 --2- 172.21.15.27:0/916762646 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5bc12c650 0x7fb5bc09b9c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:39.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.298+0000 7fb5c2072640 1 --2- 172.21.15.27:0/916762646 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fb5bc071720 0x7fb5bc09b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:39.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.298+0000 7fb5c2072640 1 -- 172.21.15.27:0/916762646 >> 172.21.15.27:0/916762646 conn(0x7fb5bc0a1e10 msgr2=0x7fb5bc12b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:39.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.298+0000 7fb5c2072640 1 -- 172.21.15.27:0/916762646 shutdown_connections 2024-09-21T03:28:39.300 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:39.298+0000 7fb5c2072640 1 -- 172.21.15.27:0/916762646 wait complete. 2024-09-21T03:28:39.711 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-09-21T03:28:39.710+0000 7f3bf5a0b640 -1 osd.2 23 log_to_monitors true 2024-09-21T03:28:39.835 INFO:tasks.ceph.osd.2.smithi027.stderr:2024-09-21T03:28:39.833+0000 7f3be90f4640 -1 osd.2 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T03:28:40.989 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-09-21T03:28:40.988+0000 7f32c2e14640 -1 osd.3 24 log_to_monitors true 2024-09-21T03:28:41.848 INFO:tasks.ceph.osd.3.smithi027.stderr:2024-09-21T03:28:41.847+0000 7f32b658f640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T03:28:42.311 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:28:42.441 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.440+0000 7fad83cb6640 1 -- 172.21.15.27:0/2547701740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad7c12c820 msgr2=0x7fad7c137a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:42.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.440+0000 7fad83cb6640 1 --2- 172.21.15.27:0/2547701740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad7c12c820 0x7fad7c137a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fad74005fd0 tx=0x7fad7402a820 comp rx=0 tx=0).stop 2024-09-21T03:28:42.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.440+0000 7fad83cb6640 1 -- 172.21.15.27:0/2547701740 shutdown_connections 2024-09-21T03:28:42.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.440+0000 7fad83cb6640 1 --2- 172.21.15.27:0/2547701740 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad7c12c820 0x7fad7c137a10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:42.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.440+0000 7fad83cb6640 1 --2- 172.21.15.27:0/2547701740 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fad7c071720 0x7fad7c12fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:42.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.440+0000 7fad83cb6640 1 -- 172.21.15.27:0/2547701740 >> 172.21.15.27:0/2547701740 conn(0x7fad7c0a1e10 msgr2=0x7fad7c0a2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:42.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.440+0000 7fad83cb6640 1 -- 172.21.15.27:0/2547701740 shutdown_connections 2024-09-21T03:28:42.442 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.440+0000 7fad83cb6640 1 -- 172.21.15.27:0/2547701740 wait complete. 2024-09-21T03:28:42.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.441+0000 7fad83cb6640 1 Processor -- start 2024-09-21T03:28:42.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.441+0000 7fad83cb6640 1 -- start start 2024-09-21T03:28:42.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad83cb6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad7c071720 0x7fad7c131ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:42.445 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad83cb6640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fad7c12c820 0x7fad7c131ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad83cb6640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fad7c130000 0x7fad7c132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad83cb6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c138740 con 0x7fad7c071720 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad83cb6640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fad7c1385c0 con 0x7fad7c130000 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad83cb6640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fad7c1388c0 con 0x7fad7c12c820 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad81a52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad7c071720 0x7fad7c131ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad82253640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fad7c130000 0x7fad7c132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad81a52640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad7c071720 0x7fad7c131ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37758/0 (socket says 172.21.15.27:37758) 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad81a52640 1 -- 172.21.15.27:0/1713451929 learned_addr learned my addr 172.21.15.27:0/1713451929 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad81251640 1 --2- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fad7c12c820 0x7fad7c131ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad81a52640 1 -- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fad7c12c820 msgr2=0x7fad7c131ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad81a52640 1 --2- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fad7c12c820 0x7fad7c131ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:42.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad81a52640 1 -- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fad7c130000 msgr2=0x7fad7c132530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad81a52640 1 --2- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fad7c130000 0x7fad7c132530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad81a52640 1 -- 172.21.15.27:0/1713451929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7c133cf0 con 0x7fad7c071720 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad82253640 1 --2- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fad7c130000 0x7fad7c132530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad81a52640 1 --2- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad7c071720 0x7fad7c131ab0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fad7000e910 tx=0x7fad7000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.442+0000 7fad6e7fc640 1 -- 172.21.15.27:0/1713451929 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7000cd20 con 0x7fad7c071720 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.443+0000 7fad6e7fc640 1 -- 172.21.15.27:0/1713451929 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fad70010430 con 0x7fad7c071720 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.443+0000 7fad83cb6640 1 -- 172.21.15.27:0/1713451929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad7c134b40 con 0x7fad7c071720 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.443+0000 7fad6e7fc640 1 -- 172.21.15.27:0/1713451929 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70010860 con 0x7fad7c071720 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.443+0000 7fad83cb6640 1 -- 172.21.15.27:0/1713451929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad7c1cbbf0 con 0x7fad7c071720 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.444+0000 7fad83cb6640 1 -- 172.21.15.27:0/1713451929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad7c12fac0 con 0x7fad7c071720 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.444+0000 7fad6e7fc640 1 -- 172.21.15.27:0/1713451929 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7fad7001f020 con 0x7fad7c071720 2024-09-21T03:28:42.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.444+0000 7fad6e7fc640 1 -- 172.21.15.27:0/1713451929 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 3027+0+0 (secure 0 0 0) 0x7fad700696f0 con 0x7fad7c071720 2024-09-21T03:28:42.449 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.448+0000 7fad6e7fc640 1 -- 172.21.15.27:0/1713451929 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fad70009490 con 0x7fad7c071720 2024-09-21T03:28:42.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.608+0000 7fad83cb6640 1 -- 172.21.15.27:0/1713451929 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad7c06dda0 con 0x7fad7c071720 2024-09-21T03:28:42.610 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.609+0000 7fad6e7fc640 1 -- 172.21.15.27:0/1713451929 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+9163 (secure 0 0 0) 0x7fad700687a0 con 0x7fad7c071720 2024-09-21T03:28:42.611 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:42.611 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:28:41.833949+0000","last_up_change":"2024-09-21T03:28:40.825989+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"squid","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T03:27:11.408521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":21,"up_from":27,"up_thru":27,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":1914045614}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":1914045614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":1914045614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":1914045614}]},"public_addr":"172.21.15.27:6801/1914045614","cluster_addr":"172.21.15.27:6802/1914045614","heartbeat_back_addr":"172.21.15.27:6804/1914045614","heartbeat_front_addr":"172.21.15.27:6803/1914045614","state":["exists","up"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":22,"up_from":28,"up_thru":28,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":580113737}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":580113737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":580113737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":580113737}]},"public_addr":"172.21.15.27:6805/580113737","cluster_addr":"172.21.15.27:6806/580113737","heartbeat_back_addr":"172.21.15.27:6808/580113737","heartbeat_front_addr":"172.21.15.27:6807/580113737","state":["exists","up"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":23,"up_from":29,"up_thru":29,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6809","nonce":2016211836}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6810","nonce":2016211836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6812","nonce":2016211836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6811","nonce":2016211836}]},"public_addr":"172.21.15.27:6809/2016211836","cluster_addr":"172.21.15.27:6810/2016211836","heartbeat_back_addr":"172.21.15.27:6812/2016211836","heartbeat_front_addr":"172.21.15.27:6811/2016211836","state":["exists","up"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":24,"down_at":25,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1661572621}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1661572621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1661572621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1661572621}]},"public_addr":"172.21.15.27:6813/1661572621","cluster_addr":"172.21.15.27:6814/1661572621","heartbeat_back_addr":"172.21.15.27:6816/1661572621","heartbeat_front_addr":"172.21.15.27:6815/1661572621","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":22},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":23},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":24},{"osd":3,"down_stamp":"2024-09-21T03:28:35.686744+0000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1436650201":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3459873364":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3140689679":"2024-09-22T03:28:28.273799+0000","172.21.15.27:6800/1098903051":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/1012963840":"2024-09-22T03:28:28.273799+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:28:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.611+0000 7fad83cb6640 1 -- 172.21.15.27:0/1713451929 >> v1:172.21.15.27:6800/2405211194 conn(0x7fad600514b0 legacy=0x7fad600539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.611+0000 7fad83cb6640 1 -- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad7c071720 msgr2=0x7fad7c131ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.611+0000 7fad83cb6640 1 --2- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad7c071720 0x7fad7c131ab0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fad7000e910 tx=0x7fad7000edd0 comp rx=0 tx=0).stop 2024-09-21T03:28:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.611+0000 7fad83cb6640 1 -- 172.21.15.27:0/1713451929 shutdown_connections 2024-09-21T03:28:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.611+0000 7fad83cb6640 1 --2- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fad7c130000 0x7fad7c132530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.611+0000 7fad83cb6640 1 --2- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fad7c12c820 0x7fad7c131ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.611+0000 7fad83cb6640 1 --2- 172.21.15.27:0/1713451929 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fad7c071720 0x7fad7c131ab0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.611+0000 7fad83cb6640 1 -- 172.21.15.27:0/1713451929 >> 172.21.15.27:0/1713451929 conn(0x7fad7c0a1e10 msgr2=0x7fad7c09c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:42.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.611+0000 7fad83cb6640 1 -- 172.21.15.27:0/1713451929 shutdown_connections 2024-09-21T03:28:42.614 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:42.611+0000 7fad83cb6640 1 -- 172.21.15.27:0/1713451929 wait complete. 2024-09-21T03:28:45.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:28:45.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.753+0000 7f19c71d2640 1 -- 172.21.15.27:0/351506950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c0071a30 msgr2=0x7f19c0070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:45.754 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.753+0000 7f19c71d2640 1 --2- 172.21.15.27:0/351506950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c0071a30 0x7f19c0070cb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f19b4009860 tx=0x7f19b402a380 comp rx=0 tx=0).stop 2024-09-21T03:28:45.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.753+0000 7f19c71d2640 1 -- 172.21.15.27:0/351506950 shutdown_connections 2024-09-21T03:28:45.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.753+0000 7f19c71d2640 1 --2- 172.21.15.27:0/351506950 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f19c0130280 0x7f19c012c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:45.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.753+0000 7f19c71d2640 1 --2- 172.21.15.27:0/351506950 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f19c012f900 0x7f19c012c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:45.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.753+0000 7f19c71d2640 1 --2- 172.21.15.27:0/351506950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c0071a30 0x7f19c0070cb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:45.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.753+0000 7f19c71d2640 1 -- 172.21.15.27:0/351506950 >> 172.21.15.27:0/351506950 conn(0x7f19c00a2050 msgr2=0x7f19c00a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:45.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.753+0000 7f19c71d2640 1 -- 172.21.15.27:0/351506950 shutdown_connections 2024-09-21T03:28:45.755 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.753+0000 7f19c71d2640 1 -- 172.21.15.27:0/351506950 wait complete. 2024-09-21T03:28:45.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.754+0000 7f19c71d2640 1 Processor -- start 2024-09-21T03:28:45.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.754+0000 7f19c71d2640 1 -- start start 2024-09-21T03:28:45.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c71d2640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f19c0071a30 0x7f19c009b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:45.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c71d2640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c012f900 0x7f19c009b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:45.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c71d2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f19c0130280 0x7f19c009bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:45.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c71d2640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f19c01373d0 con 0x7f19c012f900 2024-09-21T03:28:45.756 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c71d2640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f19c0137250 con 0x7f19c0130280 2024-09-21T03:28:45.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c71d2640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f19c0137550 con 0x7f19c0071a30 2024-09-21T03:28:45.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c012f900 0x7f19c009b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:45.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c4f6e640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f19c0071a30 0x7f19c009b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:45.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19bffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c012f900 0x7f19c009b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37766/0 (socket says 172.21.15.27:37766) 2024-09-21T03:28:45.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c4f6e640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f19c0071a30 0x7f19c009b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:50402/0 (socket says 172.21.15.27:50402) 2024-09-21T03:28:45.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19bffff640 1 -- 172.21.15.27:0/876378493 learned_addr learned my addr 172.21.15.27:0/876378493 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:45.757 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c576f640 1 --2- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f19c0130280 0x7f19c009bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c4f6e640 1 -- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f19c0130280 msgr2=0x7f19c009bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:45.758 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c4f6e640 1 --2- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f19c0130280 0x7f19c009bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c4f6e640 1 -- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c012f900 msgr2=0x7f19c009b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c4f6e640 1 --2- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c012f900 0x7f19c009b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c4f6e640 1 -- 172.21.15.27:0/876378493 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19c0134220 con 0x7f19c0071a30 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19c576f640 1 --2- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f19c0130280 0x7f19c009bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.755+0000 7f19bffff640 1 --2- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c012f900 0x7f19c009b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.756+0000 7f19c4f6e640 1 --2- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f19c0071a30 0x7f19c009b480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f19c012ffb0 tx=0x7f19b4033b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.756+0000 7f19bd7fa640 1 -- 172.21.15.27:0/876378493 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b403f020 con 0x7f19c0071a30 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.756+0000 7f19bd7fa640 1 -- 172.21.15.27:0/876378493 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f19b4033d60 con 0x7f19c0071a30 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.756+0000 7f19bd7fa640 1 -- 172.21.15.27:0/876378493 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4037040 con 0x7f19c0071a30 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.756+0000 7f19c71d2640 1 -- 172.21.15.27:0/876378493 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19c0134da0 con 0x7f19c0071a30 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.756+0000 7f19c71d2640 1 -- 172.21.15.27:0/876378493 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f19c01352b0 con 0x7f19c0071a30 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.757+0000 7f19c71d2640 1 -- 172.21.15.27:0/876378493 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19c0134790 con 0x7f19c0071a30 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.757+0000 7f19bd7fa640 1 -- 172.21.15.27:0/876378493 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f19b4042020 con 0x7f19c0071a30 2024-09-21T03:28:45.759 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.758+0000 7f19bd7fa640 1 -- 172.21.15.27:0/876378493 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(32..32 src has 1..32) ==== 3027+0+0 (secure 0 0 0) 0x7f19b4088dd0 con 0x7f19c0071a30 2024-09-21T03:28:45.763 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.761+0000 7f19bd7fa640 1 -- 172.21.15.27:0/876378493 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f19b4049030 con 0x7f19c0071a30 2024-09-21T03:28:45.923 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.922+0000 7f19c71d2640 1 -- 172.21.15.27:0/876378493 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19c0070cb0 con 0x7f19c0071a30 2024-09-21T03:28:45.924 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.923+0000 7f19bd7fa640 1 -- 172.21.15.27:0/876378493 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+9171 (secure 0 0 0) 0x7f19b408a070 con 0x7f19c0071a30 2024-09-21T03:28:45.924 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:45.924 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":32,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:28:43.842077+0000","last_up_change":"2024-09-21T03:28:42.840845+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"squid","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T03:27:11.408521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":21,"up_from":27,"up_thru":27,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":1914045614}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":1914045614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":1914045614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":1914045614}]},"public_addr":"172.21.15.27:6801/1914045614","cluster_addr":"172.21.15.27:6802/1914045614","heartbeat_back_addr":"172.21.15.27:6804/1914045614","heartbeat_front_addr":"172.21.15.27:6803/1914045614","state":["exists","up"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":22,"up_from":28,"up_thru":28,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":580113737}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":580113737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":580113737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":580113737}]},"public_addr":"172.21.15.27:6805/580113737","cluster_addr":"172.21.15.27:6806/580113737","heartbeat_back_addr":"172.21.15.27:6808/580113737","heartbeat_front_addr":"172.21.15.27:6807/580113737","state":["exists","up"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":23,"up_from":29,"up_thru":31,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6809","nonce":2016211836}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6810","nonce":2016211836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6812","nonce":2016211836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6811","nonce":2016211836}]},"public_addr":"172.21.15.27:6809/2016211836","cluster_addr":"172.21.15.27:6810/2016211836","heartbeat_back_addr":"172.21.15.27:6812/2016211836","heartbeat_front_addr":"172.21.15.27:6811/2016211836","state":["exists","up"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":24,"up_from":31,"up_thru":31,"down_at":25,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1188450781}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1188450781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1188450781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1188450781}]},"public_addr":"172.21.15.27:6813/1188450781","cluster_addr":"172.21.15.27:6814/1188450781","heartbeat_back_addr":"172.21.15.27:6816/1188450781","heartbeat_front_addr":"172.21.15.27:6815/1188450781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":22},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":23},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":24},{"osd":3,"down_stamp":"2024-09-21T03:28:35.686744+0000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1436650201":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3459873364":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3140689679":"2024-09-22T03:28:28.273799+0000","172.21.15.27:6800/1098903051":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/1012963840":"2024-09-22T03:28:28.273799+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:28:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.925+0000 7f19c71d2640 1 -- 172.21.15.27:0/876378493 >> v1:172.21.15.27:6800/2405211194 conn(0x7f19a40514b0 legacy=0x7f19a40539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.925+0000 7f19c71d2640 1 -- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f19c0071a30 msgr2=0x7f19c009b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.925+0000 7f19c71d2640 1 --2- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f19c0071a30 0x7f19c009b480 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f19c012ffb0 tx=0x7f19b4033b70 comp rx=0 tx=0).stop 2024-09-21T03:28:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.925+0000 7f19c71d2640 1 -- 172.21.15.27:0/876378493 shutdown_connections 2024-09-21T03:28:45.926 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.925+0000 7f19c71d2640 1 --2- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f19c0130280 0x7f19c009bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:45.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.925+0000 7f19c71d2640 1 --2- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19c012f900 0x7f19c009b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:45.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.925+0000 7f19c71d2640 1 --2- 172.21.15.27:0/876378493 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f19c0071a30 0x7f19c009b480 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:45.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.925+0000 7f19c71d2640 1 -- 172.21.15.27:0/876378493 >> 172.21.15.27:0/876378493 conn(0x7f19c00a2050 msgr2=0x7f19c012ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:45.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.925+0000 7f19c71d2640 1 -- 172.21.15.27:0/876378493 shutdown_connections 2024-09-21T03:28:45.927 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:45.925+0000 7f19c71d2640 1 -- 172.21.15.27:0/876378493 wait complete. 2024-09-21T03:28:45.937 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T03:28:45.938 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T03:28:45.952 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T03:28:45.952 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:28:45.952 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph versions' 2024-09-21T03:28:46.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.063+0000 7f872949b640 1 -- 172.21.15.27:0/3660417100 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f872412fe40 msgr2=0x7f8724070390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.063+0000 7f872949b640 1 --2- 172.21.15.27:0/3660417100 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f872412fe40 0x7f8724070390 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f870c009860 tx=0x7f870c02a450 comp rx=0 tx=0).stop 2024-09-21T03:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.063+0000 7f872949b640 1 -- 172.21.15.27:0/3660417100 shutdown_connections 2024-09-21T03:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.063+0000 7f872949b640 1 --2- 172.21.15.27:0/3660417100 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f872412fe40 0x7f8724070390 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.063+0000 7f872949b640 1 --2- 172.21.15.27:0/3660417100 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f872412f4f0 0x7f872412f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.063+0000 7f872949b640 1 -- 172.21.15.27:0/3660417100 >> 172.21.15.27:0/3660417100 conn(0x7f872409edd0 msgr2=0x7f872409f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.063+0000 7f872949b640 1 -- 172.21.15.27:0/3660417100 shutdown_connections 2024-09-21T03:28:46.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.063+0000 7f872949b640 1 -- 172.21.15.27:0/3660417100 wait complete. 2024-09-21T03:28:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.064+0000 7f872949b640 1 Processor -- start 2024-09-21T03:28:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.064+0000 7f872949b640 1 -- start start 2024-09-21T03:28:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f872949b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87240708d0 0x7f8724130610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:46.067 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f872949b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f872412f4f0 0x7f8724130b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f872949b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f872412fe40 0x7f8724131090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f872949b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8724138350 con 0x7f87240708d0 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f872949b640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f87241381d0 con 0x7f872412fe40 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f872949b640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f87241384d0 con 0x7f872412f4f0 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f871a7fc640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f872412f4f0 0x7f8724130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f8722ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87240708d0 0x7f8724130610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f871a7fc640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f872412f4f0 0x7f8724130b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:50418/0 (socket says 172.21.15.27:50418) 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f871a7fc640 1 -- 172.21.15.27:0/4036473254 learned_addr learned my addr 172.21.15.27:0/4036473254 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f87237fe640 1 --2- 172.21.15.27:0/4036473254 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f872412fe40 0x7f8724131090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f8722ffd640 1 -- 172.21.15.27:0/4036473254 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f872412f4f0 msgr2=0x7f8724130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f8722ffd640 1 --2- 172.21.15.27:0/4036473254 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f872412f4f0 0x7f8724130b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.068 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f8722ffd640 1 -- 172.21.15.27:0/4036473254 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f872412fe40 msgr2=0x7f8724131090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f8722ffd640 1 --2- 172.21.15.27:0/4036473254 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f872412fe40 0x7f8724131090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f8722ffd640 1 -- 172.21.15.27:0/4036473254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8724133940 con 0x7f87240708d0 2024-09-21T03:28:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.065+0000 7f8722ffd640 1 --2- 172.21.15.27:0/4036473254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87240708d0 0x7f8724130610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f871400e9f0 tx=0x7f871400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.066+0000 7f871bfff640 1 -- 172.21.15.27:0/4036473254 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871400cd80 con 0x7f87240708d0 2024-09-21T03:28:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.066+0000 7f871bfff640 1 -- 172.21.15.27:0/4036473254 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8714004020 con 0x7f87240708d0 2024-09-21T03:28:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.066+0000 7f872949b640 1 -- 172.21.15.27:0/4036473254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8724133630 con 0x7f87240708d0 2024-09-21T03:28:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.066+0000 7f871bfff640 1 -- 172.21.15.27:0/4036473254 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8714004450 con 0x7f87240708d0 2024-09-21T03:28:46.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.066+0000 7f872949b640 1 -- 172.21.15.27:0/4036473254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87241cba00 con 0x7f87240708d0 2024-09-21T03:28:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.067+0000 7f872949b640 1 -- 172.21.15.27:0/4036473254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f872412f900 con 0x7f87240708d0 2024-09-21T03:28:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.067+0000 7f871bfff640 1 -- 172.21.15.27:0/4036473254 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f87140035f0 con 0x7f87240708d0 2024-09-21T03:28:46.070 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.067+0000 7f871bfff640 1 -- 172.21.15.27:0/4036473254 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 3027+0+0 (secure 0 0 0) 0x7f8714069880 con 0x7f87240708d0 2024-09-21T03:28:46.072 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.071+0000 7f871bfff640 1 -- 172.21.15.27:0/4036473254 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f8714068930 con 0x7f87240708d0 2024-09-21T03:28:46.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.286+0000 7f872949b640 1 -- 172.21.15.27:0/4036473254 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "versions"} v 0) -- 0x7f8724128680 con 0x7f87240708d0 2024-09-21T03:28:46.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.287+0000 7f871bfff640 1 -- 172.21.15.27:0/4036473254 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) ==== 56+0+499 (secure 0 0 0) 0x7f8714009370 con 0x7f87240708d0 2024-09-21T03:28:46.288 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-21T03:28:46.288 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": { 2024-09-21T03:28:46.288 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)": 3 2024-09-21T03:28:46.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.289 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": { 2024-09-21T03:28:46.289 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)": 2 2024-09-21T03:28:46.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.289 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": { 2024-09-21T03:28:46.289 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)": 4 2024-09-21T03:28:46.289 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.289 INFO:teuthology.orchestra.run.smithi027.stdout: "overall": { 2024-09-21T03:28:46.289 INFO:teuthology.orchestra.run.smithi027.stdout: "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)": 9 2024-09-21T03:28:46.289 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.289 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-21T03:28:46.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.289+0000 7f872949b640 1 -- 172.21.15.27:0/4036473254 >> v1:172.21.15.27:6800/2405211194 conn(0x7f87040514b0 legacy=0x7f87040539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.289+0000 7f872949b640 1 -- 172.21.15.27:0/4036473254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87240708d0 msgr2=0x7f8724130610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.289+0000 7f872949b640 1 --2- 172.21.15.27:0/4036473254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87240708d0 0x7f8724130610 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f871400e9f0 tx=0x7f871400eeb0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.289+0000 7f872949b640 1 -- 172.21.15.27:0/4036473254 shutdown_connections 2024-09-21T03:28:46.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.289+0000 7f872949b640 1 --2- 172.21.15.27:0/4036473254 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f872412fe40 0x7f8724131090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.290 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.289+0000 7f872949b640 1 --2- 172.21.15.27:0/4036473254 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f872412f4f0 0x7f8724130b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.289+0000 7f872949b640 1 --2- 172.21.15.27:0/4036473254 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87240708d0 0x7f8724130610 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.289+0000 7f872949b640 1 -- 172.21.15.27:0/4036473254 >> 172.21.15.27:0/4036473254 conn(0x7f872409edd0 msgr2=0x7f872409ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.289+0000 7f872949b640 1 -- 172.21.15.27:0/4036473254 shutdown_connections 2024-09-21T03:28:46.291 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.289+0000 7f872949b640 1 -- 172.21.15.27:0/4036473254 wait complete. 2024-09-21T03:28:46.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph osd dump -f json-pretty' 2024-09-21T03:28:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.414+0000 7f33166e1640 1 -- 172.21.15.27:0/2114258790 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f331012ac00 msgr2=0x7f3310132860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.414+0000 7f33166e1640 1 --2- 172.21.15.27:0/2114258790 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f331012ac00 0x7f3310132860 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3304009830 tx=0x7f330402a480 comp rx=0 tx=0).stop 2024-09-21T03:28:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.414+0000 7f33166e1640 1 -- 172.21.15.27:0/2114258790 shutdown_connections 2024-09-21T03:28:46.415 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.414+0000 7f33166e1640 1 --2- 172.21.15.27:0/2114258790 >> v2:172.21.15.27:3300/0 conn(0x7f3310132da0 0x7f33101351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.414+0000 7f33166e1640 1 --2- 172.21.15.27:0/2114258790 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f331012ac00 0x7f3310132860 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.414+0000 7f33166e1640 1 -- 172.21.15.27:0/2114258790 >> 172.21.15.27:0/2114258790 conn(0x7f3310073d10 msgr2=0x7f3310074120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.414+0000 7f33166e1640 1 -- 172.21.15.27:0/2114258790 shutdown_connections 2024-09-21T03:28:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.414+0000 7f33166e1640 1 -- 172.21.15.27:0/2114258790 wait complete. 2024-09-21T03:28:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.415+0000 7f33166e1640 1 Processor -- start 2024-09-21T03:28:46.416 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.415+0000 7f33166e1640 1 -- start start 2024-09-21T03:28:46.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f33166e1640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f331012a0e0 0x7f331012de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:46.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f33166e1640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3310132da0 0x7f331012e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:46.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f33166e1640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33101305a0 0x7f331012e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:46.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f33166e1640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f33101365c0 con 0x7f33101305a0 2024-09-21T03:28:46.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f33166e1640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3310136440 con 0x7f331012a0e0 2024-09-21T03:28:46.418 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f33166e1640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3310136740 con 0x7f3310132da0 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330ffff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f331012a0e0 0x7f331012de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f3314c7e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33101305a0 0x7f331012e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330f7fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3310132da0 0x7f331012e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330ffff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f331012a0e0 0x7f331012de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40072/0 (socket says 172.21.15.27:40072) 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330ffff640 1 -- 172.21.15.27:0/1411242835 learned_addr learned my addr 172.21.15.27:0/1411242835 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330ffff640 1 -- 172.21.15.27:0/1411242835 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3310132da0 msgr2=0x7f331012e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330ffff640 1 --2- 172.21.15.27:0/1411242835 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3310132da0 0x7f331012e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330ffff640 1 -- 172.21.15.27:0/1411242835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33101305a0 msgr2=0x7f331012e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330ffff640 1 --2- 172.21.15.27:0/1411242835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33101305a0 0x7f331012e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330ffff640 1 -- 172.21.15.27:0/1411242835 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33101300b0 con 0x7f331012a0e0 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f3314c7e640 1 --2- 172.21.15.27:0/1411242835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33101305a0 0x7f331012e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330ffff640 1 --2- 172.21.15.27:0/1411242835 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f331012a0e0 0x7f331012de40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f330000c400 tx=0x7f330000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:46.419 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330cff9640 1 -- 172.21.15.27:0/1411242835 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300011790 con 0x7f331012a0e0 2024-09-21T03:28:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f330cff9640 1 -- 172.21.15.27:0/1411242835 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3300017070 con 0x7f331012a0e0 2024-09-21T03:28:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.416+0000 7f33166e1640 1 -- 172.21.15.27:0/1411242835 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33101c99d0 con 0x7f331012a0e0 2024-09-21T03:28:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.417+0000 7f330cff9640 1 -- 172.21.15.27:0/1411242835 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3300011ca0 con 0x7f331012a0e0 2024-09-21T03:28:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.417+0000 7f33166e1640 1 -- 172.21.15.27:0/1411242835 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f33101c9d80 con 0x7f331012a0e0 2024-09-21T03:28:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.417+0000 7f33166e1640 1 -- 172.21.15.27:0/1411242835 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f331012a4f0 con 0x7f331012a0e0 2024-09-21T03:28:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.418+0000 7f330cff9640 1 -- 172.21.15.27:0/1411242835 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f330001c460 con 0x7f331012a0e0 2024-09-21T03:28:46.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.418+0000 7f330cff9640 1 -- 172.21.15.27:0/1411242835 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(32..32 src has 1..32) ==== 3027+0+0 (secure 0 0 0) 0x7f3300068d30 con 0x7f331012a0e0 2024-09-21T03:28:46.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.422+0000 7f330cff9640 1 -- 172.21.15.27:0/1411242835 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f3300067de0 con 0x7f331012a0e0 2024-09-21T03:28:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.582+0000 7f33166e1640 1 -- 172.21.15.27:0/1411242835 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f3310125f70 con 0x7f331012a0e0 2024-09-21T03:28:46.584 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.583+0000 7f330cff9640 1 -- 172.21.15.27:0/1411242835 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v32) ==== 81+0+16835 (secure 0 0 0) 0x7f3300011e40 con 0x7f331012a0e0 2024-09-21T03:28:46.585 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:46.585 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-21T03:28:46.585 INFO:teuthology.orchestra.run.smithi027.stdout: "epoch": 32, 2024-09-21T03:28:46.585 INFO:teuthology.orchestra.run.smithi027.stdout: "fsid": "22a90953-a6e7-4790-ab76-aa25cec033b2", 2024-09-21T03:28:46.585 INFO:teuthology.orchestra.run.smithi027.stdout: "created": "2024-09-21T03:26:54.150622+0000", 2024-09-21T03:28:46.585 INFO:teuthology.orchestra.run.smithi027.stdout: "modified": "2024-09-21T03:28:43.842077+0000", 2024-09-21T03:28:46.585 INFO:teuthology.orchestra.run.smithi027.stdout: "last_up_change": "2024-09-21T03:28:42.840845+0000", 2024-09-21T03:28:46.585 INFO:teuthology.orchestra.run.smithi027.stdout: "last_in_change": "2024-09-21T03:26:56.875414+0000", 2024-09-21T03:28:46.585 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_num": 5799936, 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_set": [ 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "pglog_hardlimit", 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "purged_snapdirs", 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "recovery_deletes", 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "sortbitwise" 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_version": 9, 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_snapshot": "", 2024-09-21T03:28:46.586 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_max": 2, 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "max_osd": 4, 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "require_min_compat_client": "squid", 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "min_compat_client": "jewel", 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "require_osd_release": "squid", 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "allow_crimson": false, 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "pools": [ 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 1, 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": ".mgr", 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-09-21T03:27:07.545484+0000", 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 1, 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool", 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 2, 2024-09-21T03:28:46.587 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 1, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "is_stretch_pool": false, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 1, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 1, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 1, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 1, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 1, 2024-09-21T03:28:46.588 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "14", 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-09-21T03:28:46.589 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 0, 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 0, 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T03:28:46.590 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-09-21T03:28:46.591 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "options": { 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_max": 32, 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_min": 1 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": {} 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.592 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 4, 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 4, 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 0.5, 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2, 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2, 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 1, 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 1, 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 1 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 2, 2024-09-21T03:28:46.593 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_name": "rbd", 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "create_time": "2024-09-21T03:27:11.408521+0000", 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "flags": 8193, 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "type": 1, 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "size": 2, 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "min_size": 1, 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_rule": 0, 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_count": 0, 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_target": 0, 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "is_stretch_pool": false, 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "object_hash": 2, 2024-09-21T03:28:46.594 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_autoscale_mode": "off", 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num": 8, 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num": 8, 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_placement_num_target": 8, 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_target": 8, 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_num_pending": 8, 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "last_pg_merge_meta": { 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "source_pgid": "0.0", 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "ready_epoch": 0, 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_started": 0, 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "last_epoch_clean": 0, 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "source_version": "0'0", 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "target_version": "0'0" 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.595 INFO:teuthology.orchestra.run.smithi027.stdout: "last_change": "18", 2024-09-21T03:28:46.596 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend": "0", 2024-09-21T03:28:46.596 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T03:28:46.596 INFO:teuthology.orchestra.run.smithi027.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T03:28:46.596 INFO:teuthology.orchestra.run.smithi027.stdout: "auid": 0, 2024-09-21T03:28:46.596 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_mode": "selfmanaged", 2024-09-21T03:28:46.596 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_seq": 2, 2024-09-21T03:28:46.596 INFO:teuthology.orchestra.run.smithi027.stdout: "snap_epoch": 18, 2024-09-21T03:28:46.596 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_snaps": [], 2024-09-21T03:28:46.596 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps": "[]", 2024-09-21T03:28:46.596 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_bytes": 0, 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "quota_max_objects": 0, 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "tiers": [], 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "tier_of": -1, 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "read_tier": -1, 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "write_tier": -1, 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_mode": "none", 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_bytes": 0, 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "target_max_objects": 0, 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_flush_age": 0, 2024-09-21T03:28:46.597 INFO:teuthology.orchestra.run.smithi027.stdout: "cache_min_evict_age": 0, 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profile": "", 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_params": { 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "none" 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_period": 0, 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_count": 0, 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "use_gmt_hitset": true, 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "min_read_recency_for_promote": 0, 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "min_write_recency_for_promote": 0, 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "hit_set_search_last_n": 0, 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "grade_table": [], 2024-09-21T03:28:46.598 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_width": 0, 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "expected_num_objects": 0, 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "fast_read": false, 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "options": {}, 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "application_metadata": { 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "rbd": {} 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "read_balance": { 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "score_type": "Fair distribution", 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "score_acting": 2.5, 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "score_stable": 2.5, 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "optimal_score": 1, 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_acting": 2.5, 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "raw_score_stable": 2.5, 2024-09-21T03:28:46.599 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity_weighted": 1, 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity": 1, 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: "average_primary_affinity_weighted": 1 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: "osds": [ 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "5bfee00c-9ea5-41bd-b128-5eb75667f4f1", 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 11, 2024-09-21T03:28:46.600 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 21, 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 27, 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 27, 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 22, 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6801", 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1914045614 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.601 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6802", 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1914045614 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6804", 2024-09-21T03:28:46.602 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1914045614 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6803", 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1914045614 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6801/1914045614", 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6802/1914045614", 2024-09-21T03:28:46.603 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6804/1914045614", 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6803/1914045614", 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "121ea55e-e6d7-4d7e-bbc8-ef3fac12060f", 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 9, 2024-09-21T03:28:46.604 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 22, 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 28, 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 28, 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 23, 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6805", 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 580113737 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.605 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6806", 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 580113737 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.606 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6808", 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 580113737 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6807", 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 580113737 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6805/580113737", 2024-09-21T03:28:46.607 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6806/580113737", 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6808/580113737", 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6807/580113737", 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "b0834a51-8a1a-4bdf-9e82-9b77d445c2a5", 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-21T03:28:46.608 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 11, 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 23, 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 29, 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 31, 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 24, 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6809", 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2016211836 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.609 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6810", 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2016211836 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.610 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.611 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6812", 2024-09-21T03:28:46.611 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2016211836 2024-09-21T03:28:46.611 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.611 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.611 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.611 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-21T03:28:46.611 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.611 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.611 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.613 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6811", 2024-09-21T03:28:46.613 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 2016211836 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6809/2016211836", 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6810/2016211836", 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6812/2016211836", 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6811/2016211836", 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.614 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "uuid": "344f8d01-27ba-4598-a138-5fdde9141b60", 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "up": 1, 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "in": 1, 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "weight": 1, 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_affinity": 1, 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_begin": 11, 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "last_clean_end": 24, 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "up_from": 31, 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "up_thru": 31, 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "down_at": 25, 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "lost_at": 0, 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addrs": { 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.615 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6813", 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1188450781 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addrs": { 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6814", 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1188450781 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.616 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addrs": { 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6816", 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1188450781 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addrs": { 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: "addrvec": [ 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.617 INFO:teuthology.orchestra.run.smithi027.stdout: "type": "v1", 2024-09-21T03:28:46.618 INFO:teuthology.orchestra.run.smithi027.stdout: "addr": "172.21.15.27:6815", 2024-09-21T03:28:46.618 INFO:teuthology.orchestra.run.smithi027.stdout: "nonce": 1188450781 2024-09-21T03:28:46.618 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.618 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.618 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.618 INFO:teuthology.orchestra.run.smithi027.stdout: "public_addr": "172.21.15.27:6813/1188450781", 2024-09-21T03:28:46.618 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_addr": "172.21.15.27:6814/1188450781", 2024-09-21T03:28:46.618 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_back_addr": "172.21.15.27:6816/1188450781", 2024-09-21T03:28:46.618 INFO:teuthology.orchestra.run.smithi027.stdout: "heartbeat_front_addr": "172.21.15.27:6815/1188450781", 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: "state": [ 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: "exists", 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: "up" 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: "osd_xinfo": [ 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 0, 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-21T03:28:46.619 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-21T03:28:46.620 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-21T03:28:46.620 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-21T03:28:46.620 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-21T03:27:03.620971+0000", 2024-09-21T03:28:46.621 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 22 2024-09-21T03:28:46.621 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.621 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.621 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 1, 2024-09-21T03:28:46.621 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-21T03:28:46.621 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-21T03:28:46.621 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-21T03:27:03.459037+0000", 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 23 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 2, 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "0.000000", 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-21T03:28:46.622 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-21T03:27:03.899158+0000", 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 24 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: { 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": 3, 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: "down_stamp": "2024-09-21T03:28:35.686744+0000", 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_probability": 0, 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: "laggy_interval": 0, 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: "features": 4540719139924082687, 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: "old_weight": 0, 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: "last_purged_snaps_scrub": "2024-09-21T03:27:04.132502+0000", 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: "dead_epoch": 0 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap": [], 2024-09-21T03:28:46.623 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_items": [], 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_upmap_primaries": [], 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "pg_temp": [], 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "primary_temp": [], 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "blocklist": { 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/1436650201": "2024-09-22T03:28:28.273799+0000", 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/3459873364": "2024-09-22T03:28:28.273799+0000", 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/3140689679": "2024-09-22T03:28:28.273799+0000", 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:6800/1098903051": "2024-09-22T03:28:28.273799+0000", 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "172.21.15.27:0/1012963840": "2024-09-22T03:28:28.273799+0000" 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "range_blocklist": {}, 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "erasure_code_profiles": { 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "default": { 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "crush-failure-domain": "osd", 2024-09-21T03:28:46.624 INFO:teuthology.orchestra.run.smithi027.stdout: "k": "2", 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "m": "1", 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "plugin": "jerasure", 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "technique": "reed_sol_van" 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "removed_snaps_queue": [], 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "new_removed_snaps": [], 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "new_purged_snaps": [], 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "crush_node_flags": {}, 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "device_class_flags": {}, 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode": { 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_enabled": false, 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_bucket_count": 0, 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "degraded_stretch_mode": 0, 2024-09-21T03:28:46.625 INFO:teuthology.orchestra.run.smithi027.stdout: "recovering_stretch_mode": 0, 2024-09-21T03:28:46.626 INFO:teuthology.orchestra.run.smithi027.stdout: "stretch_mode_bucket": 0 2024-09-21T03:28:46.626 INFO:teuthology.orchestra.run.smithi027.stdout: } 2024-09-21T03:28:46.626 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-21T03:28:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.585+0000 7f33166e1640 1 -- 172.21.15.27:0/1411242835 >> v1:172.21.15.27:6800/2405211194 conn(0x7f32fc0514b0 legacy=0x7f32fc0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.585+0000 7f33166e1640 1 -- 172.21.15.27:0/1411242835 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f331012a0e0 msgr2=0x7f331012de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.585+0000 7f33166e1640 1 --2- 172.21.15.27:0/1411242835 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f331012a0e0 0x7f331012de40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f330000c400 tx=0x7f330000c8c0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.627 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.585+0000 7f33166e1640 1 -- 172.21.15.27:0/1411242835 shutdown_connections 2024-09-21T03:28:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.585+0000 7f33166e1640 1 --2- 172.21.15.27:0/1411242835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33101305a0 0x7f331012e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.585+0000 7f33166e1640 1 --2- 172.21.15.27:0/1411242835 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3310132da0 0x7f331012e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.585+0000 7f33166e1640 1 --2- 172.21.15.27:0/1411242835 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f331012a0e0 0x7f331012de40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.585+0000 7f33166e1640 1 -- 172.21.15.27:0/1411242835 >> 172.21.15.27:0/1411242835 conn(0x7f3310073d10 msgr2=0x7f331009cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.585+0000 7f33166e1640 1 -- 172.21.15.27:0/1411242835 shutdown_connections 2024-09-21T03:28:46.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.585+0000 7f33166e1640 1 -- 172.21.15.27:0/1411242835 wait complete. 2024-09-21T03:28:46.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs dump' 2024-09-21T03:28:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.739+0000 7f56f7026640 1 -- 172.21.15.27:0/390314870 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f56f0070aa0 msgr2=0x7f56f0070eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.739+0000 7f56f7026640 1 --2- 172.21.15.27:0/390314870 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f56f0070aa0 0x7f56f0070eb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f56e8005fd0 tx=0x7f56e802a820 comp rx=0 tx=0).stop 2024-09-21T03:28:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.739+0000 7f56f7026640 1 -- 172.21.15.27:0/390314870 shutdown_connections 2024-09-21T03:28:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.739+0000 7f56f7026640 1 --2- 172.21.15.27:0/390314870 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f56f0070aa0 0x7f56f0070eb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.739+0000 7f56f7026640 1 --2- 172.21.15.27:0/390314870 >> v2:172.21.15.27:3300/0 conn(0x7f56f012f4f0 0x7f56f012f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.741 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.739+0000 7f56f7026640 1 -- 172.21.15.27:0/390314870 >> 172.21.15.27:0/390314870 conn(0x7f56f009edd0 msgr2=0x7f56f009f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.739+0000 7f56f7026640 1 -- 172.21.15.27:0/390314870 shutdown_connections 2024-09-21T03:28:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.739+0000 7f56f7026640 1 -- 172.21.15.27:0/390314870 wait complete. 2024-09-21T03:28:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.741+0000 7f56f7026640 1 Processor -- start 2024-09-21T03:28:46.742 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.741+0000 7f56f7026640 1 -- start start 2024-09-21T03:28:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.741+0000 7f56f7026640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f56f0070aa0 0x7f56f012ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.741+0000 7f56f7026640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f56f012f4f0 0x7f56f012b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.741+0000 7f56f7026640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f012fe40 0x7f56f012b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.741+0000 7f56f7026640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f56f0138250 con 0x7f56f012fe40 2024-09-21T03:28:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.741+0000 7f56f7026640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f56f01380d0 con 0x7f56f0070aa0 2024-09-21T03:28:46.744 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.741+0000 7f56f7026640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f56f01383d0 con 0x7f56f012f4f0 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f4dc2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f56f0070aa0 0x7f56f012ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56effff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f56f012f4f0 0x7f56f012b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f55c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f012fe40 0x7f56f012b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f55c3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f012fe40 0x7f56f012b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37802/0 (socket says 172.21.15.27:37802) 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56effff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f56f012f4f0 0x7f56f012b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:50430/0 (socket says 172.21.15.27:50430) 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f4dc2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f56f0070aa0 0x7f56f012ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40082/0 (socket says 172.21.15.27:40082) 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f55c3640 1 -- 172.21.15.27:0/3323895148 learned_addr learned my addr 172.21.15.27:0/3323895148 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f55c3640 1 -- 172.21.15.27:0/3323895148 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f56f012f4f0 msgr2=0x7f56f012b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f55c3640 1 --2- 172.21.15.27:0/3323895148 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f56f012f4f0 0x7f56f012b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f55c3640 1 -- 172.21.15.27:0/3323895148 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f56f0070aa0 msgr2=0x7f56f012ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f55c3640 1 --2- 172.21.15.27:0/3323895148 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f56f0070aa0 0x7f56f012ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f55c3640 1 -- 172.21.15.27:0/3323895148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f0129f60 con 0x7f56f012fe40 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56effff640 1 --2- 172.21.15.27:0/3323895148 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f56f012f4f0 0x7f56f012b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:46.745 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f55c3640 1 --2- 172.21.15.27:0/3323895148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f012fe40 0x7f56f012b6d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f56e8035040 tx=0x7f56e8032c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56ed7fa640 1 -- 172.21.15.27:0/3323895148 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e8037040 con 0x7f56f012fe40 2024-09-21T03:28:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56ed7fa640 1 -- 172.21.15.27:0/3323895148 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f56e8042070 con 0x7f56f012fe40 2024-09-21T03:28:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f7026640 1 -- 172.21.15.27:0/3323895148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f012cb30 con 0x7f56f012fe40 2024-09-21T03:28:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56ed7fa640 1 -- 172.21.15.27:0/3323895148 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e8003e60 con 0x7f56f012fe40 2024-09-21T03:28:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.742+0000 7f56f7026640 1 -- 172.21.15.27:0/3323895148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f01cb8f0 con 0x7f56f012fe40 2024-09-21T03:28:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.743+0000 7f56f7026640 1 -- 172.21.15.27:0/3323895148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f009e550 con 0x7f56f012fe40 2024-09-21T03:28:46.746 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.744+0000 7f56ed7fa640 1 -- 172.21.15.27:0/3323895148 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f56e8004260 con 0x7f56f012fe40 2024-09-21T03:28:46.747 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.744+0000 7f56ed7fa640 1 -- 172.21.15.27:0/3323895148 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 3027+0+0 (secure 0 0 0) 0x7f56e8089cb0 con 0x7f56f012fe40 2024-09-21T03:28:46.749 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.748+0000 7f56ed7fa640 1 -- 172.21.15.27:0/3323895148 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f56f009e550 con 0x7f56f012fe40 2024-09-21T03:28:46.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.931+0000 7f56f7026640 1 -- 172.21.15.27:0/3323895148 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f56f0070eb0 con 0x7f56f012fe40 2024-09-21T03:28:46.932 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.931+0000 7f56ed7fa640 1 -- 172.21.15.27:0/3323895148 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 1 v1) ==== 75+0+440 (secure 0 0 0) 0x7f56e8088d60 con 0x7f56f012fe40 2024-09-21T03:28:46.933 INFO:teuthology.orchestra.run.smithi027.stdout:e1 2024-09-21T03:28:46.933 INFO:teuthology.orchestra.run.smithi027.stdout:btime 2024-09-21T03:26:54:150286+0000 2024-09-21T03:28:46.933 INFO:teuthology.orchestra.run.smithi027.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T03:28:46.933 INFO:teuthology.orchestra.run.smithi027.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T03:28:46.933 INFO:teuthology.orchestra.run.smithi027.stdout:legacy client fscid: -1 2024-09-21T03:28:46.933 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:46.933 INFO:teuthology.orchestra.run.smithi027.stdout:No filesystems configured 2024-09-21T03:28:46.933 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 1 2024-09-21T03:28:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.933+0000 7f56f7026640 1 -- 172.21.15.27:0/3323895148 >> v1:172.21.15.27:6800/2405211194 conn(0x7f56d40514e0 legacy=0x7f56d40539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.933+0000 7f56f7026640 1 -- 172.21.15.27:0/3323895148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f012fe40 msgr2=0x7f56f012b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.933+0000 7f56f7026640 1 --2- 172.21.15.27:0/3323895148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f012fe40 0x7f56f012b6d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f56e8035040 tx=0x7f56e8032c60 comp rx=0 tx=0).stop 2024-09-21T03:28:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.933+0000 7f56f7026640 1 -- 172.21.15.27:0/3323895148 shutdown_connections 2024-09-21T03:28:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.933+0000 7f56f7026640 1 --2- 172.21.15.27:0/3323895148 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56f012fe40 0x7f56f012b6d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.933+0000 7f56f7026640 1 --2- 172.21.15.27:0/3323895148 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f56f012f4f0 0x7f56f012b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.933+0000 7f56f7026640 1 --2- 172.21.15.27:0/3323895148 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f56f0070aa0 0x7f56f012ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.933+0000 7f56f7026640 1 -- 172.21.15.27:0/3323895148 >> 172.21.15.27:0/3323895148 conn(0x7f56f009edd0 msgr2=0x7f56f00975e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:46.935 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.933+0000 7f56f7026640 1 -- 172.21.15.27:0/3323895148 shutdown_connections 2024-09-21T03:28:46.936 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:46.933+0000 7f56f7026640 1 -- 172.21.15.27:0/3323895148 wait complete. 2024-09-21T03:28:46.946 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph osd require-osd-release squid' 2024-09-21T03:28:47.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.060+0000 7fddec7ae640 1 -- 172.21.15.27:0/729541149 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdde4134ef0 msgr2=0x7fdde4137310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.060+0000 7fddec7ae640 1 --2- 172.21.15.27:0/729541149 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdde4134ef0 0x7fdde4137310 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdddc005fd0 tx=0x7fdddc02aa90 comp rx=0 tx=0).stop 2024-09-21T03:28:47.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.060+0000 7fddec7ae640 1 -- 172.21.15.27:0/729541149 shutdown_connections 2024-09-21T03:28:47.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.060+0000 7fddec7ae640 1 --2- 172.21.15.27:0/729541149 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdde4134ef0 0x7fdde4137310 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.060+0000 7fddec7ae640 1 --2- 172.21.15.27:0/729541149 >> v2:172.21.15.27:3300/0 conn(0x7fdde412f7a0 0x7fdde41349b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.060+0000 7fddec7ae640 1 -- 172.21.15.27:0/729541149 >> 172.21.15.27:0/729541149 conn(0x7fdde409e6c0 msgr2=0x7fdde409ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:47.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.060+0000 7fddec7ae640 1 -- 172.21.15.27:0/729541149 shutdown_connections 2024-09-21T03:28:47.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.060+0000 7fddec7ae640 1 -- 172.21.15.27:0/729541149 wait complete. 2024-09-21T03:28:47.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.061+0000 7fddec7ae640 1 Processor -- start 2024-09-21T03:28:47.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.061+0000 7fddec7ae640 1 -- start start 2024-09-21T03:28:47.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddec7ae640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdde4071a30 0x7fdde41c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:47.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddec7ae640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fdde412f7a0 0x7fdde41c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:47.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddec7ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde4134ef0 0x7fdde41c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:47.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddec7ae640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fdde4138690 con 0x7fdde4134ef0 2024-09-21T03:28:47.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddec7ae640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fdde4138510 con 0x7fdde4071a30 2024-09-21T03:28:47.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddec7ae640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fdde4138810 con 0x7fdde412f7a0 2024-09-21T03:28:47.064 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddea54a640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdde4071a30 0x7fdde41c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fdde9d49640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fdde412f7a0 0x7fdde41c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddea54a640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdde4071a30 0x7fdde41c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40104/0 (socket says 172.21.15.27:40104) 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddea54a640 1 -- 172.21.15.27:0/437136077 learned_addr learned my addr 172.21.15.27:0/437136077 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddead4b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde4134ef0 0x7fdde41c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddea54a640 1 -- 172.21.15.27:0/437136077 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fdde412f7a0 msgr2=0x7fdde41c6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddea54a640 1 --2- 172.21.15.27:0/437136077 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fdde412f7a0 0x7fdde41c6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddea54a640 1 -- 172.21.15.27:0/437136077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde4134ef0 msgr2=0x7fdde41c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddea54a640 1 --2- 172.21.15.27:0/437136077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde4134ef0 0x7fdde41c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddea54a640 1 -- 172.21.15.27:0/437136077 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde4131d00 con 0x7fdde4071a30 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddead4b640 1 --2- 172.21.15.27:0/437136077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde4134ef0 0x7fdde41c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddea54a640 1 --2- 172.21.15.27:0/437136077 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdde4071a30 0x7fdde41c5ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdde0001eb0 tx=0x7fdde000c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fdddaffd640 1 -- 172.21.15.27:0/437136077 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0017070 con 0x7fdde4071a30 2024-09-21T03:28:47.065 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fdddaffd640 1 -- 172.21.15.27:0/437136077 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fdde00053c0 con 0x7fdde4071a30 2024-09-21T03:28:47.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.062+0000 7fddec7ae640 1 -- 172.21.15.27:0/437136077 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde4131810 con 0x7fdde4071a30 2024-09-21T03:28:47.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.063+0000 7fdddaffd640 1 -- 172.21.15.27:0/437136077 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0003830 con 0x7fdde4071a30 2024-09-21T03:28:47.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.063+0000 7fddec7ae640 1 -- 172.21.15.27:0/437136077 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde4130ee0 con 0x7fdde4071a30 2024-09-21T03:28:47.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.064+0000 7fddec7ae640 1 -- 172.21.15.27:0/437136077 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdde412fcd0 con 0x7fdde4071a30 2024-09-21T03:28:47.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.064+0000 7fdddaffd640 1 -- 172.21.15.27:0/437136077 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7fdde001a070 con 0x7fdde4071a30 2024-09-21T03:28:47.066 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.064+0000 7fdddaffd640 1 -- 172.21.15.27:0/437136077 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(32..32 src has 1..32) ==== 3027+0+0 (secure 0 0 0) 0x7fdde00683c0 con 0x7fdde4071a30 2024-09-21T03:28:47.069 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.068+0000 7fdddaffd640 1 -- 172.21.15.27:0/437136077 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fdde0067470 con 0x7fdde4071a30 2024-09-21T03:28:47.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.235+0000 7fddec7ae640 1 -- 172.21.15.27:0/437136077 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd require-osd-release", "release": "squid"} v 0) -- 0x7fdde406e0e0 con 0x7fdde4071a30 2024-09-21T03:28:47.237 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.236+0000 7fdddaffd640 1 -- 172.21.15.27:0/437136077 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd require-osd-release", "release": "squid"}]=0 v32) ==== 91+0+0 (secure 0 0 0) 0x7fdde0066fb0 con 0x7fdde4071a30 2024-09-21T03:28:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.238+0000 7fddec7ae640 1 -- 172.21.15.27:0/437136077 >> v1:172.21.15.27:6800/2405211194 conn(0x7fddc80514b0 legacy=0x7fddc80539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.238+0000 7fddec7ae640 1 -- 172.21.15.27:0/437136077 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdde4071a30 msgr2=0x7fdde41c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.238+0000 7fddec7ae640 1 --2- 172.21.15.27:0/437136077 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdde4071a30 0x7fdde41c5ae0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdde0001eb0 tx=0x7fdde000c890 comp rx=0 tx=0).stop 2024-09-21T03:28:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.238+0000 7fddec7ae640 1 -- 172.21.15.27:0/437136077 shutdown_connections 2024-09-21T03:28:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.238+0000 7fddec7ae640 1 --2- 172.21.15.27:0/437136077 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdde4134ef0 0x7fdde41c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.238+0000 7fddec7ae640 1 --2- 172.21.15.27:0/437136077 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fdde412f7a0 0x7fdde41c6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.238+0000 7fddec7ae640 1 --2- 172.21.15.27:0/437136077 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fdde4071a30 0x7fdde41c5ae0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.238+0000 7fddec7ae640 1 -- 172.21.15.27:0/437136077 >> 172.21.15.27:0/437136077 conn(0x7fdde409e6c0 msgr2=0x7fdde40727c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.238+0000 7fddec7ae640 1 -- 172.21.15.27:0/437136077 shutdown_connections 2024-09-21T03:28:47.240 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.238+0000 7fddec7ae640 1 -- 172.21.15.27:0/437136077 wait complete. 2024-09-21T03:28:47.251 DEBUG:teuthology.orchestra.run.smithi027:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'for f in `ceph osd pool ls` ; do ceph osd pool set $f pg_autoscale_mode off ; done' 2024-09-21T03:28:47.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.362+0000 7f3a7c49b640 1 -- 172.21.15.27:0/299990378 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a7412fcb0 msgr2=0x7f3a741300c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.362+0000 7f3a7c49b640 1 --2- 172.21.15.27:0/299990378 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a7412fcb0 0x7f3a741300c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c005fd0 tx=0x7f3a6c02a6e0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.363 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.362+0000 7f3a7c49b640 1 -- 172.21.15.27:0/299990378 shutdown_connections 2024-09-21T03:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.362+0000 7f3a7c49b640 1 --2- 172.21.15.27:0/299990378 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a7412fcb0 0x7f3a741300c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.362+0000 7f3a7c49b640 1 --2- 172.21.15.27:0/299990378 >> v2:172.21.15.27:3300/0 conn(0x7f3a7412cd70 0x7f3a7412f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.362+0000 7f3a7c49b640 1 --2- 172.21.15.27:0/299990378 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a7412c420 0x7f3a7412c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.362+0000 7f3a7c49b640 1 -- 172.21.15.27:0/299990378 >> 172.21.15.27:0/299990378 conn(0x7f3a7409e8d0 msgr2=0x7f3a7409ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.362+0000 7f3a7c49b640 1 -- 172.21.15.27:0/299990378 shutdown_connections 2024-09-21T03:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.362+0000 7f3a7c49b640 1 -- 172.21.15.27:0/299990378 wait complete. 2024-09-21T03:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.363+0000 7f3a7c49b640 1 Processor -- start 2024-09-21T03:28:47.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.363+0000 7f3a7c49b640 1 -- start start 2024-09-21T03:28:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7c49b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a7412c420 0x7f3a74134b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7c49b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a7412cd70 0x7f3a74135080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7c49b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a7412fcb0 0x7f3a741355c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7c49b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3a74137610 con 0x7f3a7412cd70 2024-09-21T03:28:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7c49b640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3a74137490 con 0x7f3a7412fcb0 2024-09-21T03:28:47.366 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7c49b640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3a74137790 con 0x7f3a7412c420 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7a237640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a7412c420 0x7f3a74134b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7a237640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a7412c420 0x7f3a74134b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:50452/0 (socket says 172.21.15.27:50452) 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7a237640 1 -- 172.21.15.27:0/3456632656 learned_addr learned my addr 172.21.15.27:0/3456632656 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a79a36640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a7412cd70 0x7f3a74135080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7aa38640 1 --2- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a7412fcb0 0x7f3a741355c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7a237640 1 -- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a7412fcb0 msgr2=0x7f3a741355c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7a237640 1 --2- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a7412fcb0 0x7f3a741355c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7a237640 1 -- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a7412cd70 msgr2=0x7f3a74135080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7a237640 1 --2- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a7412cd70 0x7f3a74135080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7a237640 1 -- 172.21.15.27:0/3456632656 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a74135b00 con 0x7f3a7412c420 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7aa38640 1 --2- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a7412fcb0 0x7f3a741355c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a79a36640 1 --2- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a7412cd70 0x7f3a74135080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a7a237640 1 --2- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a7412c420 0x7f3a74134b40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000e9c0 tx=0x7f3a7000edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:47.367 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.364+0000 7f3a6affd640 1 -- 172.21.15.27:0/3456632656 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7000cd20 con 0x7f3a7412c420 2024-09-21T03:28:47.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.365+0000 7f3a6affd640 1 -- 172.21.15.27:0/3456632656 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3a70010430 con 0x7f3a7412c420 2024-09-21T03:28:47.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.365+0000 7f3a7c49b640 1 -- 172.21.15.27:0/3456632656 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a74134570 con 0x7f3a7412c420 2024-09-21T03:28:47.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.365+0000 7f3a6affd640 1 -- 172.21.15.27:0/3456632656 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a70010880 con 0x7f3a7412c420 2024-09-21T03:28:47.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.365+0000 7f3a7c49b640 1 -- 172.21.15.27:0/3456632656 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a741cbaa0 con 0x7f3a7412c420 2024-09-21T03:28:47.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.365+0000 7f3a7c49b640 1 -- 172.21.15.27:0/3456632656 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a7412f770 con 0x7f3a7412c420 2024-09-21T03:28:47.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.366+0000 7f3a6affd640 1 -- 172.21.15.27:0/3456632656 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f3a7001f020 con 0x7f3a7412c420 2024-09-21T03:28:47.368 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.366+0000 7f3a6affd640 1 -- 172.21.15.27:0/3456632656 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(32..32 src has 1..32) ==== 3027+0+0 (secure 0 0 0) 0x7f3a70069700 con 0x7f3a7412c420 2024-09-21T03:28:47.371 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.370+0000 7f3a6affd640 1 -- 172.21.15.27:0/3456632656 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f3a700687b0 con 0x7f3a7412c420 2024-09-21T03:28:47.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.528+0000 7f3a7c49b640 1 -- 172.21.15.27:0/3456632656 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd pool ls"} v 0) -- 0x7f3a7412b7a0 con 0x7f3a7412c420 2024-09-21T03:28:47.530 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.529+0000 7f3a6affd640 1 -- 172.21.15.27:0/3456632656 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd pool ls"}]=0 v32) ==== 59+0+9 (secure 0 0 0) 0x7f3a70009400 con 0x7f3a7412c420 2024-09-21T03:28:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.531+0000 7f3a7c49b640 1 -- 172.21.15.27:0/3456632656 >> v1:172.21.15.27:6800/2405211194 conn(0x7f3a580514b0 legacy=0x7f3a580539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.531+0000 7f3a7c49b640 1 -- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a7412c420 msgr2=0x7f3a74134b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.531+0000 7f3a7c49b640 1 --2- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a7412c420 0x7f3a74134b40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000e9c0 tx=0x7f3a7000edc0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.531+0000 7f3a7c49b640 1 -- 172.21.15.27:0/3456632656 shutdown_connections 2024-09-21T03:28:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.531+0000 7f3a7c49b640 1 --2- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3a7412fcb0 0x7f3a741355c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.531+0000 7f3a7c49b640 1 --2- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3a7412cd70 0x7f3a74135080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.531+0000 7f3a7c49b640 1 --2- 172.21.15.27:0/3456632656 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3a7412c420 0x7f3a74134b40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.531+0000 7f3a7c49b640 1 -- 172.21.15.27:0/3456632656 >> 172.21.15.27:0/3456632656 conn(0x7f3a7409e8d0 msgr2=0x7f3a740705b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.531+0000 7f3a7c49b640 1 -- 172.21.15.27:0/3456632656 shutdown_connections 2024-09-21T03:28:47.533 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.531+0000 7f3a7c49b640 1 -- 172.21.15.27:0/3456632656 wait complete. 2024-09-21T03:28:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.632+0000 7f1d67bae640 1 -- 172.21.15.27:0/1492914346 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1d6012d300 msgr2=0x7f1d6012d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.632+0000 7f1d67bae640 1 --2- 172.21.15.27:0/1492914346 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1d6012d300 0x7f1d6012d710 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1d50009830 tx=0x7f1d5002a360 comp rx=0 tx=0).stop 2024-09-21T03:28:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.632+0000 7f1d67bae640 1 -- 172.21.15.27:0/1492914346 shutdown_connections 2024-09-21T03:28:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.632+0000 7f1d67bae640 1 --2- 172.21.15.27:0/1492914346 >> v2:172.21.15.27:3300/0 conn(0x7f1d6012dc50 0x7f1d60134690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.632+0000 7f1d67bae640 1 --2- 172.21.15.27:0/1492914346 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1d6012d300 0x7f1d6012d710 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.632+0000 7f1d67bae640 1 --2- 172.21.15.27:0/1492914346 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1d6012a190 0x7f1d6012a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.632+0000 7f1d67bae640 1 -- 172.21.15.27:0/1492914346 >> 172.21.15.27:0/1492914346 conn(0x7f1d60073950 msgr2=0x7f1d60073d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.632+0000 7f1d67bae640 1 -- 172.21.15.27:0/1492914346 shutdown_connections 2024-09-21T03:28:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.632+0000 7f1d67bae640 1 -- 172.21.15.27:0/1492914346 wait complete. 2024-09-21T03:28:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.633+0000 7f1d67bae640 1 Processor -- start 2024-09-21T03:28:47.636 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d67bae640 1 -- start start 2024-09-21T03:28:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d67bae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d6012a190 0x7f1d601337f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d67bae640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1d6012d300 0x7f1d60133d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d67bae640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1d6012dc50 0x7f1d601c5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d67bae640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f1d60135120 con 0x7f1d6012a190 2024-09-21T03:28:47.637 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d67bae640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f1d60134fa0 con 0x7f1d6012d300 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d67bae640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f1d601352a0 con 0x7f1d6012dc50 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d6594a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d6012a190 0x7f1d601337f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d6594a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d6012a190 0x7f1d601337f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37860/0 (socket says 172.21.15.27:37860) 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d65149640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1d6012d300 0x7f1d60133d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d6594a640 1 -- 172.21.15.27:0/2474118815 learned_addr learned my addr 172.21.15.27:0/2474118815 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.634+0000 7f1d6614b640 1 --2- 172.21.15.27:0/2474118815 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1d6012dc50 0x7f1d601c5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.635+0000 7f1d65149640 1 -- 172.21.15.27:0/2474118815 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1d6012dc50 msgr2=0x7f1d601c5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.635+0000 7f1d65149640 1 --2- 172.21.15.27:0/2474118815 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1d6012dc50 0x7f1d601c5680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.635+0000 7f1d65149640 1 -- 172.21.15.27:0/2474118815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d6012a190 msgr2=0x7f1d601337f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.635+0000 7f1d65149640 1 --2- 172.21.15.27:0/2474118815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d6012a190 0x7f1d601337f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.635+0000 7f1d65149640 1 -- 172.21.15.27:0/2474118815 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d60131e30 con 0x7f1d6012d300 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.635+0000 7f1d65149640 1 --2- 172.21.15.27:0/2474118815 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1d6012d300 0x7f1d60133d30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d50031000 tx=0x7f1d50033b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:47.638 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.635+0000 7f1d567fc640 1 -- 172.21.15.27:0/2474118815 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5003f020 con 0x7f1d6012d300 2024-09-21T03:28:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.635+0000 7f1d567fc640 1 -- 172.21.15.27:0/2474118815 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1d50033d60 con 0x7f1d6012d300 2024-09-21T03:28:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.635+0000 7f1d67bae640 1 -- 172.21.15.27:0/2474118815 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d60132c30 con 0x7f1d6012d300 2024-09-21T03:28:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.635+0000 7f1d567fc640 1 -- 172.21.15.27:0/2474118815 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d50037040 con 0x7f1d6012d300 2024-09-21T03:28:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.635+0000 7f1d67bae640 1 -- 172.21.15.27:0/2474118815 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d60134270 con 0x7f1d6012d300 2024-09-21T03:28:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.636+0000 7f1d567fc640 1 -- 172.21.15.27:0/2474118815 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f1d50042020 con 0x7f1d6012d300 2024-09-21T03:28:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.636+0000 7f1d67bae640 1 -- 172.21.15.27:0/2474118815 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d6012a5a0 con 0x7f1d6012d300 2024-09-21T03:28:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.637+0000 7f1d567fc640 1 -- 172.21.15.27:0/2474118815 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(32..32 src has 1..32) ==== 3027+0+0 (secure 0 0 0) 0x7f1d50088e60 con 0x7f1d6012d300 2024-09-21T03:28:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.640+0000 7f1d567fc640 1 -- 172.21.15.27:0/2474118815 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f1d50049030 con 0x7f1d6012d300 2024-09-21T03:28:47.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.801+0000 7f1d67bae640 1 -- 172.21.15.27:0/2474118815 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_autoscale_mode", "val": "off"} v 0) -- 0x7f1d600697c0 con 0x7f1d6012d300 2024-09-21T03:28:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.982+0000 7f1d567fc640 1 -- 172.21.15.27:0/2474118815 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_autoscale_mode", "val": "off"}]=0 set pool 1 pg_autoscale_mode to off v33) ==== 153+0+0 (secure 0 0 0) 0x7f1d50087f10 con 0x7f1d6012d300 2024-09-21T03:28:47.984 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 1 pg_autoscale_mode to off 2024-09-21T03:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.985+0000 7f1d67bae640 1 -- 172.21.15.27:0/2474118815 >> v1:172.21.15.27:6800/2405211194 conn(0x7f1d440514b0 legacy=0x7f1d440539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.985+0000 7f1d67bae640 1 -- 172.21.15.27:0/2474118815 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1d6012d300 msgr2=0x7f1d60133d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.985+0000 7f1d67bae640 1 --2- 172.21.15.27:0/2474118815 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1d6012d300 0x7f1d60133d30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1d50031000 tx=0x7f1d50033b70 comp rx=0 tx=0).stop 2024-09-21T03:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.985+0000 7f1d67bae640 1 -- 172.21.15.27:0/2474118815 shutdown_connections 2024-09-21T03:28:47.986 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.985+0000 7f1d67bae640 1 --2- 172.21.15.27:0/2474118815 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f1d6012dc50 0x7f1d601c5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.985+0000 7f1d67bae640 1 --2- 172.21.15.27:0/2474118815 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f1d6012d300 0x7f1d60133d30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.985+0000 7f1d67bae640 1 --2- 172.21.15.27:0/2474118815 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1d6012a190 0x7f1d601337f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.985+0000 7f1d67bae640 1 -- 172.21.15.27:0/2474118815 >> 172.21.15.27:0/2474118815 conn(0x7f1d60073950 msgr2=0x7f1d60124ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.985+0000 7f1d67bae640 1 -- 172.21.15.27:0/2474118815 shutdown_connections 2024-09-21T03:28:47.987 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:47.985+0000 7f1d67bae640 1 -- 172.21.15.27:0/2474118815 wait complete. 2024-09-21T03:28:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.087+0000 7f91df146640 1 -- 172.21.15.27:0/1997396938 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f91d812a280 msgr2=0x7f91d812a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.087+0000 7f91df146640 1 --2- 172.21.15.27:0/1997396938 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f91d812a280 0x7f91d812a690 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f91d0009be0 tx=0x7f91d002a800 comp rx=0 tx=0).stop 2024-09-21T03:28:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.087+0000 7f91df146640 1 -- 172.21.15.27:0/1997396938 shutdown_connections 2024-09-21T03:28:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.087+0000 7f91df146640 1 --2- 172.21.15.27:0/1997396938 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f91d812d3b0 0x7f91d812d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.087+0000 7f91df146640 1 --2- 172.21.15.27:0/1997396938 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f91d812a280 0x7f91d812a690 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.087+0000 7f91df146640 1 -- 172.21.15.27:0/1997396938 >> 172.21.15.27:0/1997396938 conn(0x7f91d8073950 msgr2=0x7f91d8073d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.087+0000 7f91df146640 1 -- 172.21.15.27:0/1997396938 shutdown_connections 2024-09-21T03:28:48.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.087+0000 7f91df146640 1 -- 172.21.15.27:0/1997396938 wait complete. 2024-09-21T03:28:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.088+0000 7f91df146640 1 Processor -- start 2024-09-21T03:28:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.088+0000 7f91df146640 1 -- start start 2024-09-21T03:28:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91df146640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f91d812a280 0x7f91d8131930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:48.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91df146640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d812d3b0 0x7f91d8131e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91df146640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f91d812dd00 0x7f91d81323b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91df146640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f91d81364c0 con 0x7f91d812d3b0 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91df146640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f91d8136340 con 0x7f91d812a280 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91df146640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f91d8136640 con 0x7f91d812dd00 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91dcee2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f91d812a280 0x7f91d8131930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91d7fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d812d3b0 0x7f91d8131e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91dcee2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f91d812a280 0x7f91d8131930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40174/0 (socket says 172.21.15.27:40174) 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91dd6e3640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f91d812dd00 0x7f91d81323b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91dcee2640 1 -- 172.21.15.27:0/3435169729 learned_addr learned my addr 172.21.15.27:0/3435169729 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91dcee2640 1 -- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f91d812dd00 msgr2=0x7f91d81323b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91dcee2640 1 --2- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f91d812dd00 0x7f91d81323b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91dcee2640 1 -- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d812d3b0 msgr2=0x7f91d8131e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91dcee2640 1 --2- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d812d3b0 0x7f91d8131e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:48.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91dcee2640 1 -- 172.21.15.27:0/3435169729 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91d8134f90 con 0x7f91d812a280 2024-09-21T03:28:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91dd6e3640 1 --2- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f91d812dd00 0x7f91d81323b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.089+0000 7f91d7fff640 1 --2- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d812d3b0 0x7f91d8131e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.090+0000 7f91dcee2640 1 --2- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f91d812a280 0x7f91d8131930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f91d0006010 tx=0x7f91d0033dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.090+0000 7f91d57fa640 1 -- 172.21.15.27:0/3435169729 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0042020 con 0x7f91d812a280 2024-09-21T03:28:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.090+0000 7f91d57fa640 1 -- 172.21.15.27:0/3435169729 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f91d0004130 con 0x7f91d812a280 2024-09-21T03:28:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.090+0000 7f91df146640 1 -- 172.21.15.27:0/3435169729 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91d8134370 con 0x7f91d812a280 2024-09-21T03:28:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.090+0000 7f91d57fa640 1 -- 172.21.15.27:0/3435169729 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0037040 con 0x7f91d812a280 2024-09-21T03:28:48.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.090+0000 7f91df146640 1 -- 172.21.15.27:0/3435169729 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f91d8123950 con 0x7f91d812a280 2024-09-21T03:28:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.091+0000 7f91df146640 1 -- 172.21.15.27:0/3435169729 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91d8134980 con 0x7f91d812a280 2024-09-21T03:28:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.091+0000 7f91d57fa640 1 -- 172.21.15.27:0/3435169729 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f91d0045020 con 0x7f91d812a280 2024-09-21T03:28:48.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.091+0000 7f91d57fa640 1 -- 172.21.15.27:0/3435169729 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(33..33 src has 1..33) ==== 3027+0+0 (secure 0 0 0) 0x7f91d0089e10 con 0x7f91d812a280 2024-09-21T03:28:48.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.095+0000 7f91d57fa640 1 -- 172.21.15.27:0/3435169729 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f91d004c030 con 0x7f91d812a280 2024-09-21T03:28:48.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.252+0000 7f91df146640 1 -- 172.21.15.27:0/3435169729 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "rbd", "var": "pg_autoscale_mode", "val": "off"} v 0) -- 0x7f91d812a690 con 0x7f91d812a280 2024-09-21T03:28:48.998 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:48.997+0000 7f91d57fa640 1 -- 172.21.15.27:0/3435169729 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "rbd", "var": "pg_autoscale_mode", "val": "off"}]=0 set pool 2 pg_autoscale_mode to off v34) ==== 152+0+0 (secure 0 0 0) 0x7f91d002f070 con 0x7f91d812a280 2024-09-21T03:28:48.998 INFO:teuthology.orchestra.run.smithi027.stderr:set pool 2 pg_autoscale_mode to off 2024-09-21T03:28:49.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.000+0000 7f91df146640 1 -- 172.21.15.27:0/3435169729 >> v1:172.21.15.27:6800/2405211194 conn(0x7f91b80514b0 legacy=0x7f91b80539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.000+0000 7f91df146640 1 -- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f91d812a280 msgr2=0x7f91d8131930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.000+0000 7f91df146640 1 --2- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f91d812a280 0x7f91d8131930 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f91d0006010 tx=0x7f91d0033dd0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.000+0000 7f91df146640 1 -- 172.21.15.27:0/3435169729 shutdown_connections 2024-09-21T03:28:49.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.000+0000 7f91df146640 1 --2- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f91d812dd00 0x7f91d81323b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.000+0000 7f91df146640 1 --2- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91d812d3b0 0x7f91d8131e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.000+0000 7f91df146640 1 --2- 172.21.15.27:0/3435169729 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f91d812a280 0x7f91d8131930 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.001 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.000+0000 7f91df146640 1 -- 172.21.15.27:0/3435169729 >> 172.21.15.27:0/3435169729 conn(0x7f91d8073950 msgr2=0x7f91d8124de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:49.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.000+0000 7f91df146640 1 -- 172.21.15.27:0/3435169729 shutdown_connections 2024-09-21T03:28:49.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.000+0000 7f91df146640 1 -- 172.21.15.27:0/3435169729 wait complete. 2024-09-21T03:28:49.015 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-21T03:28:49.030 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T03:28:49.031 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T03:28:49.031 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T03:28:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.153+0000 7f365064b640 1 -- 172.21.15.27:0/2195704039 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f364812fc70 msgr2=0x7f3648130080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.153+0000 7f365064b640 1 --2- 172.21.15.27:0/2195704039 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f364812fc70 0x7f3648130080 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3644009830 tx=0x7f364402a480 comp rx=0 tx=0).stop 2024-09-21T03:28:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.153+0000 7f365064b640 1 -- 172.21.15.27:0/2195704039 shutdown_connections 2024-09-21T03:28:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.153+0000 7f365064b640 1 --2- 172.21.15.27:0/2195704039 >> v2:172.21.15.27:3300/0 conn(0x7f36481305c0 0x7f3648072830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.153+0000 7f365064b640 1 --2- 172.21.15.27:0/2195704039 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f364812fc70 0x7f3648130080 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.153+0000 7f365064b640 1 -- 172.21.15.27:0/2195704039 >> 172.21.15.27:0/2195704039 conn(0x7f36480a1f40 msgr2=0x7f36480a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.153+0000 7f365064b640 1 -- 172.21.15.27:0/2195704039 shutdown_connections 2024-09-21T03:28:49.155 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.153+0000 7f365064b640 1 -- 172.21.15.27:0/2195704039 wait complete. 2024-09-21T03:28:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.154+0000 7f365064b640 1 Processor -- start 2024-09-21T03:28:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.154+0000 7f365064b640 1 -- start start 2024-09-21T03:28:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f365064b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3648072d70 0x7f36481c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f365064b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f364812fc70 0x7f36481c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f365064b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36481305c0 0x7f36481c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f365064b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3648138890 con 0x7f3648072d70 2024-09-21T03:28:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f365064b640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3648138710 con 0x7f364812fc70 2024-09-21T03:28:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f365064b640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3648138a10 con 0x7f36481305c0 2024-09-21T03:28:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f364dbe6640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f364812fc70 0x7f36481c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f364e3e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3648072d70 0x7f36481c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f364dbe6640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f364812fc70 0x7f36481c6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40186/0 (socket says 172.21.15.27:40186) 2024-09-21T03:28:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f364e3e7640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3648072d70 0x7f36481c5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37892/0 (socket says 172.21.15.27:37892) 2024-09-21T03:28:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f364dbe6640 1 -- 172.21.15.27:0/2743362229 learned_addr learned my addr 172.21.15.27:0/2743362229 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.155+0000 7f364ebe8640 1 --2- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36481305c0 0x7f36481c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f364e3e7640 1 -- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36481305c0 msgr2=0x7f36481c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f364e3e7640 1 --2- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36481305c0 0x7f36481c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f364e3e7640 1 -- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f364812fc70 msgr2=0x7f36481c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f364e3e7640 1 --2- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f364812fc70 0x7f36481c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f364e3e7640 1 -- 172.21.15.27:0/2743362229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3648094740 con 0x7f3648072d70 2024-09-21T03:28:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f364dbe6640 1 --2- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f364812fc70 0x7f36481c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f364e3e7640 1 --2- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3648072d70 0x7f36481c5ce0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3644031000 tx=0x7f3644032b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f363effd640 1 -- 172.21.15.27:0/2743362229 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3644040070 con 0x7f3648072d70 2024-09-21T03:28:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f363effd640 1 -- 172.21.15.27:0/2743362229 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3644003900 con 0x7f3648072d70 2024-09-21T03:28:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f365064b640 1 -- 172.21.15.27:0/2743362229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3648094ee0 con 0x7f3648072d70 2024-09-21T03:28:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f363effd640 1 -- 172.21.15.27:0/2743362229 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3644003d50 con 0x7f3648072d70 2024-09-21T03:28:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.156+0000 7f365064b640 1 -- 172.21.15.27:0/2743362229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3648092800 con 0x7f3648072d70 2024-09-21T03:28:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.157+0000 7f365064b640 1 -- 172.21.15.27:0/2743362229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36480a08c0 con 0x7f3648072d70 2024-09-21T03:28:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.157+0000 7f363effd640 1 -- 172.21.15.27:0/2743362229 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f3644039040 con 0x7f3648072d70 2024-09-21T03:28:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.158+0000 7f363effd640 1 -- 172.21.15.27:0/2743362229 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f364408a580 con 0x7f3648072d70 2024-09-21T03:28:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.161+0000 7f363effd640 1 -- 172.21.15.27:0/2743362229 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f36480a08c0 con 0x7f3648072d70 2024-09-21T03:28:49.351 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.350+0000 7f365064b640 1 -- 172.21.15.27:0/2743362229 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3648130080 con 0x7f3648072d70 2024-09-21T03:28:49.353 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.352+0000 7f363effd640 1 -- 172.21.15.27:0/2743362229 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v9) ==== 74+0+124219 (secure 0 0 0) 0x7f3644089630 con 0x7f3648072d70 2024-09-21T03:28:49.355 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:49.356 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.354+0000 7f365064b640 1 -- 172.21.15.27:0/2743362229 >> v1:172.21.15.27:6800/2405211194 conn(0x7f362c0514b0 legacy=0x7f362c0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.354+0000 7f365064b640 1 -- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3648072d70 msgr2=0x7f36481c5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.354+0000 7f365064b640 1 --2- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3648072d70 0x7f36481c5ce0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3644031000 tx=0x7f3644032b90 comp rx=0 tx=0).stop 2024-09-21T03:28:49.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.354+0000 7f365064b640 1 -- 172.21.15.27:0/2743362229 shutdown_connections 2024-09-21T03:28:49.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.354+0000 7f365064b640 1 --2- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f36481305c0 0x7f36481c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.354+0000 7f365064b640 1 --2- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f364812fc70 0x7f36481c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.354+0000 7f365064b640 1 --2- 172.21.15.27:0/2743362229 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3648072d70 0x7f36481c5ce0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.354+0000 7f365064b640 1 -- 172.21.15.27:0/2743362229 >> 172.21.15.27:0/2743362229 conn(0x7f36480a1f40 msgr2=0x7f364809c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:49.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.355+0000 7f365064b640 1 -- 172.21.15.27:0/2743362229 shutdown_connections 2024-09-21T03:28:49.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.355+0000 7f365064b640 1 -- 172.21.15.27:0/2743362229 wait complete. 2024-09-21T03:28:49.367 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":9,"flags":0,"active_gid":14101,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6800","nonce":2405211194}]},"active_addr":"172.21.15.27:6800/2405211194","active_change":"2024-09-21T03:28:28.274008+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":14115,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":21,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":3139608787}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1442463794}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":704836387}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.27:0","nonce":1293556394}]}]} 2024-09-21T03:28:49.369 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T03:28:49.370 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T03:28:49.370 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:28:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.483+0000 7f3ca64b6640 1 -- 172.21.15.27:0/3371209097 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3ca01353f0 msgr2=0x7f3ca0137810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.483+0000 7f3ca64b6640 1 --2- 172.21.15.27:0/3371209097 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3ca01353f0 0x7f3ca0137810 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3c98005fd0 tx=0x7f3c9802a820 comp rx=0 tx=0).stop 2024-09-21T03:28:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.483+0000 7f3ca64b6640 1 -- 172.21.15.27:0/3371209097 shutdown_connections 2024-09-21T03:28:49.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.483+0000 7f3ca64b6640 1 --2- 172.21.15.27:0/3371209097 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3ca01353f0 0x7f3ca0137810 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.483+0000 7f3ca64b6640 1 --2- 172.21.15.27:0/3371209097 >> v2:172.21.15.27:3300/0 conn(0x7f3ca0130790 0x7f3ca0134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.483+0000 7f3ca64b6640 1 -- 172.21.15.27:0/3371209097 >> 172.21.15.27:0/3371209097 conn(0x7f3ca00a1f40 msgr2=0x7f3ca00a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.483+0000 7f3ca64b6640 1 -- 172.21.15.27:0/3371209097 shutdown_connections 2024-09-21T03:28:49.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.483+0000 7f3ca64b6640 1 -- 172.21.15.27:0/3371209097 wait complete. 2024-09-21T03:28:49.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.484+0000 7f3ca64b6640 1 Processor -- start 2024-09-21T03:28:49.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.484+0000 7f3ca64b6640 1 -- start start 2024-09-21T03:28:49.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca64b6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ca012fc70 0x7f3ca0128ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:49.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca64b6640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3ca0130790 0x7f3ca0129420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:49.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca64b6640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3ca01353f0 0x7f3ca0129960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:49.487 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca64b6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca0138910 con 0x7f3ca012fc70 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca64b6640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3ca0138790 con 0x7f3ca0130790 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca64b6640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3ca0138a90 con 0x7f3ca01353f0 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3c9ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ca012fc70 0x7f3ca0128ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3c9f7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3ca0130790 0x7f3ca0129420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca4a53640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3ca01353f0 0x7f3ca0129960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3c9ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ca012fc70 0x7f3ca0128ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37908/0 (socket says 172.21.15.27:37908) 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3c9f7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3ca0130790 0x7f3ca0129420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40194/0 (socket says 172.21.15.27:40194) 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3c9ffff640 1 -- 172.21.15.27:0/1270090964 learned_addr learned my addr 172.21.15.27:0/1270090964 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca4a53640 1 -- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3ca0130790 msgr2=0x7f3ca0129420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca4a53640 1 --2- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3ca0130790 0x7f3ca0129420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca4a53640 1 -- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ca012fc70 msgr2=0x7f3ca0128ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca4a53640 1 --2- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ca012fc70 0x7f3ca0128ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.488 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca4a53640 1 -- 172.21.15.27:0/1270090964 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca012bce0 con 0x7f3ca01353f0 2024-09-21T03:28:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3c9f7fe640 1 --2- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3ca0130790 0x7f3ca0129420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3c9ffff640 1 --2- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ca012fc70 0x7f3ca0128ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca4a53640 1 --2- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3ca01353f0 0x7f3ca0129960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009000 tx=0x7f3c98032be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3c9cff9640 1 -- 172.21.15.27:0/1270090964 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98037040 con 0x7f3ca01353f0 2024-09-21T03:28:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3c9cff9640 1 -- 172.21.15.27:0/1270090964 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3c98042070 con 0x7f3ca01353f0 2024-09-21T03:28:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.485+0000 7f3ca64b6640 1 -- 172.21.15.27:0/1270090964 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca012b600 con 0x7f3ca01353f0 2024-09-21T03:28:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.486+0000 7f3c9cff9640 1 -- 172.21.15.27:0/1270090964 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c98004040 con 0x7f3ca01353f0 2024-09-21T03:28:49.489 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.486+0000 7f3ca64b6640 1 -- 172.21.15.27:0/1270090964 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ca01c5d60 con 0x7f3ca01353f0 2024-09-21T03:28:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.487+0000 7f3c9cff9640 1 -- 172.21.15.27:0/1270090964 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f3c98004200 con 0x7f3ca01353f0 2024-09-21T03:28:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.487+0000 7f3ca64b6640 1 -- 172.21.15.27:0/1270090964 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ca00a08c0 con 0x7f3ca01353f0 2024-09-21T03:28:49.490 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.487+0000 7f3c9cff9640 1 -- 172.21.15.27:0/1270090964 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f3c98089bd0 con 0x7f3ca01353f0 2024-09-21T03:28:49.492 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.491+0000 7f3c9cff9640 1 -- 172.21.15.27:0/1270090964 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f3ca00a08c0 con 0x7f3ca01353f0 2024-09-21T03:28:49.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.650+0000 7f3ca64b6640 1 -- 172.21.15.27:0/1270090964 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ca0130080 con 0x7f3ca01353f0 2024-09-21T03:28:49.652 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.651+0000 7f3c9cff9640 1 -- 172.21.15.27:0/1270090964 <== mon.2 v2:172.21.15.27:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+9171 (secure 0 0 0) 0x7f3ca012bce0 con 0x7f3ca01353f0 2024-09-21T03:28:49.653 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:49.653 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:28:48.984591+0000","last_up_change":"2024-09-21T03:28:42.840845+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"squid","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T03:27:11.408521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":21,"up_from":27,"up_thru":27,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":1914045614}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":1914045614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":1914045614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":1914045614}]},"public_addr":"172.21.15.27:6801/1914045614","cluster_addr":"172.21.15.27:6802/1914045614","heartbeat_back_addr":"172.21.15.27:6804/1914045614","heartbeat_front_addr":"172.21.15.27:6803/1914045614","state":["exists","up"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":22,"up_from":28,"up_thru":28,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":580113737}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":580113737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":580113737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":580113737}]},"public_addr":"172.21.15.27:6805/580113737","cluster_addr":"172.21.15.27:6806/580113737","heartbeat_back_addr":"172.21.15.27:6808/580113737","heartbeat_front_addr":"172.21.15.27:6807/580113737","state":["exists","up"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":23,"up_from":29,"up_thru":31,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6809","nonce":2016211836}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6810","nonce":2016211836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6812","nonce":2016211836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6811","nonce":2016211836}]},"public_addr":"172.21.15.27:6809/2016211836","cluster_addr":"172.21.15.27:6810/2016211836","heartbeat_back_addr":"172.21.15.27:6812/2016211836","heartbeat_front_addr":"172.21.15.27:6811/2016211836","state":["exists","up"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":24,"up_from":31,"up_thru":31,"down_at":25,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1188450781}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1188450781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1188450781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1188450781}]},"public_addr":"172.21.15.27:6813/1188450781","cluster_addr":"172.21.15.27:6814/1188450781","heartbeat_back_addr":"172.21.15.27:6816/1188450781","heartbeat_front_addr":"172.21.15.27:6815/1188450781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":22},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":23},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":24},{"osd":3,"down_stamp":"2024-09-21T03:28:35.686744+0000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1436650201":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3459873364":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3140689679":"2024-09-22T03:28:28.273799+0000","172.21.15.27:6800/1098903051":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/1012963840":"2024-09-22T03:28:28.273799+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:28:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.653+0000 7f3ca64b6640 1 -- 172.21.15.27:0/1270090964 >> v1:172.21.15.27:6800/2405211194 conn(0x7f3c8c0514a0 legacy=0x7f3c8c0539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.653+0000 7f3ca64b6640 1 -- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3ca01353f0 msgr2=0x7f3ca0129960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.653+0000 7f3ca64b6640 1 --2- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3ca01353f0 0x7f3ca0129960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3c98009000 tx=0x7f3c98032be0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.653+0000 7f3ca64b6640 1 -- 172.21.15.27:0/1270090964 shutdown_connections 2024-09-21T03:28:49.655 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.653+0000 7f3ca64b6640 1 --2- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3ca01353f0 0x7f3ca0129960 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.653+0000 7f3ca64b6640 1 --2- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3ca0130790 0x7f3ca0129420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.656 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.653+0000 7f3ca64b6640 1 --2- 172.21.15.27:0/1270090964 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3ca012fc70 0x7f3ca0128ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.653+0000 7f3ca64b6640 1 -- 172.21.15.27:0/1270090964 >> 172.21.15.27:0/1270090964 conn(0x7f3ca00a1f40 msgr2=0x7f3ca009bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:49.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.653+0000 7f3ca64b6640 1 -- 172.21.15.27:0/1270090964 shutdown_connections 2024-09-21T03:28:49.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.653+0000 7f3ca64b6640 1 -- 172.21.15.27:0/1270090964 wait complete. 2024-09-21T03:28:49.666 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T03:28:49.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:28:49.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.780+0000 7f1994514640 1 -- 172.21.15.27:0/1266775365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f198c12c110 msgr2=0x7f198c12cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.780+0000 7f1994514640 1 --2- 172.21.15.27:0/1266775365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f198c12c110 0x7f198c12cb80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f197c005fd0 tx=0x7f197c02a4a0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.780+0000 7f1994514640 1 -- 172.21.15.27:0/1266775365 shutdown_connections 2024-09-21T03:28:49.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.780+0000 7f1994514640 1 --2- 172.21.15.27:0/1266775365 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f198c12f650 0x7f198c12fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.780+0000 7f1994514640 1 --2- 172.21.15.27:0/1266775365 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f198c12c110 0x7f198c12cb80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.780+0000 7f1994514640 1 --2- 172.21.15.27:0/1266775365 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f198c071860 0x7f198c12c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.780+0000 7f1994514640 1 -- 172.21.15.27:0/1266775365 >> 172.21.15.27:0/1266775365 conn(0x7f198c0a1e10 msgr2=0x7f198c0a2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:49.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.780+0000 7f1994514640 1 -- 172.21.15.27:0/1266775365 shutdown_connections 2024-09-21T03:28:49.782 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.780+0000 7f1994514640 1 -- 172.21.15.27:0/1266775365 wait complete. 2024-09-21T03:28:49.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.781+0000 7f1994514640 1 Processor -- start 2024-09-21T03:28:49.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.781+0000 7f1994514640 1 -- start start 2024-09-21T03:28:49.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f1994514640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f198c071860 0x7f198c1c5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:49.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f1994514640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f198c12c110 0x7f198c1c5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:49.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f1994514640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f198c12f650 0x7f198c1c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:49.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f1994514640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f198c1374a0 con 0x7f198c12c110 2024-09-21T03:28:49.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f1994514640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f198c137320 con 0x7f198c071860 2024-09-21T03:28:49.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f1994514640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f198c137620 con 0x7f198c12f650 2024-09-21T03:28:49.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f19922b0640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f198c071860 0x7f198c1c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:49.785 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f19922b0640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f198c071860 0x7f198c1c5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40204/0 (socket says 172.21.15.27:40204) 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f1992ab1640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f198c12f650 0x7f198c1c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f19922b0640 1 -- 172.21.15.27:0/1042551741 learned_addr learned my addr 172.21.15.27:0/1042551741 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f1991aaf640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f198c12c110 0x7f198c1c5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f19922b0640 1 -- 172.21.15.27:0/1042551741 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f198c12f650 msgr2=0x7f198c1c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f19922b0640 1 --2- 172.21.15.27:0/1042551741 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f198c12f650 0x7f198c1c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f19922b0640 1 -- 172.21.15.27:0/1042551741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f198c12c110 msgr2=0x7f198c1c5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f19922b0640 1 --2- 172.21.15.27:0/1042551741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f198c12c110 0x7f198c1c5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f19922b0640 1 -- 172.21.15.27:0/1042551741 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198c134f90 con 0x7f198c071860 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f1992ab1640 1 --2- 172.21.15.27:0/1042551741 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f198c12f650 0x7f198c1c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.782+0000 7f19922b0640 1 --2- 172.21.15.27:0/1042551741 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f198c071860 0x7f198c1c5970 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f198800e910 tx=0x7f198800edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.783+0000 7f1982ffd640 1 -- 172.21.15.27:0/1042551741 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f198800cd80 con 0x7f198c071860 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.783+0000 7f1982ffd640 1 -- 172.21.15.27:0/1042551741 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f1988010430 con 0x7f198c071860 2024-09-21T03:28:49.786 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.783+0000 7f1994514640 1 -- 172.21.15.27:0/1042551741 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f198c1349c0 con 0x7f198c071860 2024-09-21T03:28:49.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.783+0000 7f1982ffd640 1 -- 172.21.15.27:0/1042551741 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1988010860 con 0x7f198c071860 2024-09-21T03:28:49.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.783+0000 7f1994514640 1 -- 172.21.15.27:0/1042551741 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f198c09b480 con 0x7f198c071860 2024-09-21T03:28:49.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.784+0000 7f1994514640 1 -- 172.21.15.27:0/1042551741 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f198c0744b0 con 0x7f198c071860 2024-09-21T03:28:49.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.784+0000 7f1982ffd640 1 -- 172.21.15.27:0/1042551741 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f198801f020 con 0x7f198c071860 2024-09-21T03:28:49.787 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.784+0000 7f1982ffd640 1 -- 172.21.15.27:0/1042551741 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f1988069760 con 0x7f198c071860 2024-09-21T03:28:49.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.788+0000 7f1982ffd640 1 -- 172.21.15.27:0/1042551741 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f198c0744b0 con 0x7f198c071860 2024-09-21T03:28:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.946+0000 7f1994514640 1 -- 172.21.15.27:0/1042551741 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f198c12fa60 con 0x7f198c071860 2024-09-21T03:28:49.947 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.946+0000 7f1982ffd640 1 -- 172.21.15.27:0/1042551741 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+9171 (secure 0 0 0) 0x7f1988009490 con 0x7f198c071860 2024-09-21T03:28:49.948 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:49.948 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:28:48.984591+0000","last_up_change":"2024-09-21T03:28:42.840845+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"squid","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T03:27:11.408521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":21,"up_from":27,"up_thru":27,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":1914045614}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":1914045614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":1914045614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":1914045614}]},"public_addr":"172.21.15.27:6801/1914045614","cluster_addr":"172.21.15.27:6802/1914045614","heartbeat_back_addr":"172.21.15.27:6804/1914045614","heartbeat_front_addr":"172.21.15.27:6803/1914045614","state":["exists","up"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":22,"up_from":28,"up_thru":28,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":580113737}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":580113737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":580113737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":580113737}]},"public_addr":"172.21.15.27:6805/580113737","cluster_addr":"172.21.15.27:6806/580113737","heartbeat_back_addr":"172.21.15.27:6808/580113737","heartbeat_front_addr":"172.21.15.27:6807/580113737","state":["exists","up"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":23,"up_from":29,"up_thru":31,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6809","nonce":2016211836}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6810","nonce":2016211836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6812","nonce":2016211836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6811","nonce":2016211836}]},"public_addr":"172.21.15.27:6809/2016211836","cluster_addr":"172.21.15.27:6810/2016211836","heartbeat_back_addr":"172.21.15.27:6812/2016211836","heartbeat_front_addr":"172.21.15.27:6811/2016211836","state":["exists","up"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":24,"up_from":31,"up_thru":31,"down_at":25,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1188450781}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1188450781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1188450781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1188450781}]},"public_addr":"172.21.15.27:6813/1188450781","cluster_addr":"172.21.15.27:6814/1188450781","heartbeat_back_addr":"172.21.15.27:6816/1188450781","heartbeat_front_addr":"172.21.15.27:6815/1188450781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":22},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":23},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":24},{"osd":3,"down_stamp":"2024-09-21T03:28:35.686744+0000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1436650201":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3459873364":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3140689679":"2024-09-22T03:28:28.273799+0000","172.21.15.27:6800/1098903051":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/1012963840":"2024-09-22T03:28:28.273799+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:28:49.949 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.948+0000 7f1994514640 1 -- 172.21.15.27:0/1042551741 >> v1:172.21.15.27:6800/2405211194 conn(0x7f19700514b0 legacy=0x7f19700539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.948+0000 7f1994514640 1 -- 172.21.15.27:0/1042551741 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f198c071860 msgr2=0x7f198c1c5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.948+0000 7f1994514640 1 --2- 172.21.15.27:0/1042551741 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f198c071860 0x7f198c1c5970 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f198800e910 tx=0x7f198800edd0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.948+0000 7f1994514640 1 -- 172.21.15.27:0/1042551741 shutdown_connections 2024-09-21T03:28:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.948+0000 7f1994514640 1 --2- 172.21.15.27:0/1042551741 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f198c12f650 0x7f198c1c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.948+0000 7f1994514640 1 --2- 172.21.15.27:0/1042551741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f198c12c110 0x7f198c1c5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.948+0000 7f1994514640 1 --2- 172.21.15.27:0/1042551741 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f198c071860 0x7f198c1c5970 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.948+0000 7f1994514640 1 -- 172.21.15.27:0/1042551741 >> 172.21.15.27:0/1042551741 conn(0x7f198c0a1e10 msgr2=0x7f198c12b810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.948+0000 7f1994514640 1 -- 172.21.15.27:0/1042551741 shutdown_connections 2024-09-21T03:28:49.950 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:49.948+0000 7f1994514640 1 -- 172.21.15.27:0/1042551741 wait complete. 2024-09-21T03:28:49.961 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T03:28:49.961 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T03:28:49.962 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T03:28:49.962 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T03:28:50.089 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7fee567cc640 1 -- 172.21.15.27:0/2689298327 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fee500718e0 msgr2=0x7fee5012c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7fee567cc640 1 --2- 172.21.15.27:0/2689298327 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fee500718e0 0x7fee5012c4c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee40009830 tx=0x7fee4002a480 comp rx=0 tx=0).stop 2024-09-21T03:28:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7f2a587f6640 1 -- 172.21.15.27:0/3183781543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5012dc40 msgr2=0x7f2a50129d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7f2a587f6640 1 --2- 172.21.15.27:0/3183781543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5012dc40 0x7f2a50129d70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2a40009860 tx=0x7f2a4002a420 comp rx=0 tx=0).stop 2024-09-21T03:28:50.090 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7fee567cc640 1 -- 172.21.15.27:0/2689298327 shutdown_connections 2024-09-21T03:28:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7fee567cc640 1 --2- 172.21.15.27:0/2689298327 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fee5012fb30 0x7fee5012ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7fee567cc640 1 --2- 172.21.15.27:0/2689298327 >> v2:172.21.15.27:3300/0 conn(0x7fee5012ca00 0x7fee5012f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7fee567cc640 1 --2- 172.21.15.27:0/2689298327 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fee500718e0 0x7fee5012c4c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7fee567cc640 1 -- 172.21.15.27:0/2689298327 >> 172.21.15.27:0/2689298327 conn(0x7fee500a1d70 msgr2=0x7fee500a2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7f2a587f6640 1 -- 172.21.15.27:0/3183781543 shutdown_connections 2024-09-21T03:28:50.091 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7fee567cc640 1 -- 172.21.15.27:0/2689298327 shutdown_connections 2024-09-21T03:28:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7f2a587f6640 1 --2- 172.21.15.27:0/3183781543 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a5012a2b0 0x7f2a5012a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7f2a587f6640 1 --2- 172.21.15.27:0/3183781543 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5012dc40 0x7f2a50129d70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7f2a587f6640 1 --2- 172.21.15.27:0/3183781543 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a5012d2f0 0x7f2a5012d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7f2a587f6640 1 -- 172.21.15.27:0/3183781543 >> 172.21.15.27:0/3183781543 conn(0x7f2a50098360 msgr2=0x7f2a50098770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.092 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7fee567cc640 1 -- 172.21.15.27:0/2689298327 wait complete. 2024-09-21T03:28:50.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.088+0000 7f2a587f6640 1 -- 172.21.15.27:0/3183781543 shutdown_connections 2024-09-21T03:28:50.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.089+0000 7f2a587f6640 1 -- 172.21.15.27:0/3183781543 wait complete. 2024-09-21T03:28:50.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7fee567cc640 1 Processor -- start 2024-09-21T03:28:50.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a587f6640 1 Processor -- start 2024-09-21T03:28:50.093 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a587f6640 1 -- start start 2024-09-21T03:28:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7fee567cc640 1 -- start start 2024-09-21T03:28:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a587f6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5012a2b0 0x7f2a501c3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a587f6640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a5012d2f0 0x7f2a501c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.094 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a587f6640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a5012dc40 0x7f2a501c4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a587f6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2a50135170 con 0x7f2a5012a2b0 2024-09-21T03:28:50.095 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a587f6640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f2a50134ff0 con 0x7f2a5012d2f0 2024-09-21T03:28:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee567cc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee500718e0 0x7fee5009b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee567cc640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fee5012ca00 0x7fee5009b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee567cc640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fee5012fb30 0x7fee5009bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee567cc640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fee501374a0 con 0x7fee500718e0 2024-09-21T03:28:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee567cc640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fee50137320 con 0x7fee5012ca00 2024-09-21T03:28:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee567cc640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fee50137620 con 0x7fee5012fb30 2024-09-21T03:28:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee4f7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fee5012ca00 0x7fee5009b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee4f7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fee5012ca00 0x7fee5009b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40238/0 (socket says 172.21.15.27:40238) 2024-09-21T03:28:50.096 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee4f7fe640 1 -- 172.21.15.27:0/682399215 learned_addr learned my addr 172.21.15.27:0/682399215 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee54d69640 1 --2- 172.21.15.27:0/682399215 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fee5012fb30 0x7fee5009bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a587f6640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f2a501352f0 con 0x7f2a5012dc40 2024-09-21T03:28:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee4ffff640 1 --2- 172.21.15.27:0/682399215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee500718e0 0x7fee5009b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.097 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a56d93640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a5012dc40 0x7f2a501c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee4f7fe640 1 -- 172.21.15.27:0/682399215 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fee5012fb30 msgr2=0x7fee5009bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee4f7fe640 1 --2- 172.21.15.27:0/682399215 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fee5012fb30 0x7fee5009bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee4f7fe640 1 -- 172.21.15.27:0/682399215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee500718e0 msgr2=0x7fee5009b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee4f7fe640 1 --2- 172.21.15.27:0/682399215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fee500718e0 0x7fee5009b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee4f7fe640 1 -- 172.21.15.27:0/682399215 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee50134940 con 0x7fee5012ca00 2024-09-21T03:28:50.098 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7fee54d69640 1 --2- 172.21.15.27:0/682399215 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fee5012fb30 0x7fee5009bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T03:28:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a55d91640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a5012d2f0 0x7f2a501c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a56d93640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a5012dc40 0x7f2a501c4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:50566/0 (socket says 172.21.15.27:50566) 2024-09-21T03:28:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a56d93640 1 -- 172.21.15.27:0/3285330373 learned_addr learned my addr 172.21.15.27:0/3285330373 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.090+0000 7f2a56592640 1 --2- 172.21.15.27:0/3285330373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5012a2b0 0x7f2a501c3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7f2a55d91640 1 -- 172.21.15.27:0/3285330373 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a5012dc40 msgr2=0x7f2a501c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.099 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7f2a55d91640 1 --2- 172.21.15.27:0/3285330373 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f2a5012dc40 0x7f2a501c4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.100 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.092+0000 7fee4f7fe640 1 --2- 172.21.15.27:0/682399215 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fee5012ca00 0x7fee5009b9c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fee4400e9c0 tx=0x7fee4400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:50.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7f2a55d91640 1 -- 172.21.15.27:0/3285330373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5012a2b0 msgr2=0x7f2a501c3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7f2a55d91640 1 --2- 172.21.15.27:0/3285330373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5012a2b0 0x7f2a501c3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7f2a55d91640 1 -- 172.21.15.27:0/3285330373 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a501318e0 con 0x7f2a5012d2f0 2024-09-21T03:28:50.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7f2a56592640 1 --2- 172.21.15.27:0/3285330373 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2a5012a2b0 0x7f2a501c3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:50.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.091+0000 7f2a55d91640 1 --2- 172.21.15.27:0/3285330373 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a5012d2f0 0x7f2a501c3c50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2a500737b0 tx=0x7f2a40005d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:50.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.092+0000 7f2a46ffd640 1 -- 172.21.15.27:0/3285330373 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a40031b50 con 0x7f2a5012d2f0 2024-09-21T03:28:50.102 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.092+0000 7f2a46ffd640 1 -- 172.21.15.27:0/3285330373 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f2a40031de0 con 0x7f2a5012d2f0 2024-09-21T03:28:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.092+0000 7f2a587f6640 1 -- 172.21.15.27:0/3285330373 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a50132770 con 0x7f2a5012d2f0 2024-09-21T03:28:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.092+0000 7f2a46ffd640 1 -- 172.21.15.27:0/3285330373 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a40005280 con 0x7f2a5012d2f0 2024-09-21T03:28:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.092+0000 7fee4cff9640 1 -- 172.21.15.27:0/682399215 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4400ccc0 con 0x7fee5012ca00 2024-09-21T03:28:50.103 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.092+0000 7f2a587f6640 1 -- 172.21.15.27:0/3285330373 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a50097070 con 0x7f2a5012d2f0 2024-09-21T03:28:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.092+0000 7fee4cff9640 1 -- 172.21.15.27:0/682399215 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fee44005020 con 0x7fee5012ca00 2024-09-21T03:28:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.093+0000 7fee4cff9640 1 -- 172.21.15.27:0/682399215 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4400ce60 con 0x7fee5012ca00 2024-09-21T03:28:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.093+0000 7fee567cc640 1 -- 172.21.15.27:0/682399215 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee501342f0 con 0x7fee5012ca00 2024-09-21T03:28:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.093+0000 7fee567cc640 1 -- 172.21.15.27:0/682399215 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fee501350f0 con 0x7fee5012ca00 2024-09-21T03:28:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.093+0000 7f2a587f6640 1 -- 172.21.15.27:0/3285330373 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2a50076d50 con 0x7f2a5012d2f0 2024-09-21T03:28:50.104 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.094+0000 7fee567cc640 1 -- 172.21.15.27:0/682399215 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fee50074620 con 0x7fee5012ca00 2024-09-21T03:28:50.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.094+0000 7f2a46ffd640 1 -- 172.21.15.27:0/3285330373 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f2a40005420 con 0x7f2a5012d2f0 2024-09-21T03:28:50.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.094+0000 7f2a46ffd640 1 -- 172.21.15.27:0/3285330373 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f2a4009b500 con 0x7f2a5012d2f0 2024-09-21T03:28:50.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.094+0000 7fee4cff9640 1 -- 172.21.15.27:0/682399215 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7fee44005450 con 0x7fee5012ca00 2024-09-21T03:28:50.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.094+0000 7f2a46ffd640 1 -- 172.21.15.27:0/3285330373 --> v1:172.21.15.27:6809/2016211836 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a3405bad0 con 0x7f2a34057dd0 2024-09-21T03:28:50.105 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.094+0000 7f2a46ffd640 1 -- 172.21.15.27:0/3285330373 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f2a400a2050 con 0x7f2a5012d2f0 2024-09-21T03:28:50.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.095+0000 7fee4cff9640 1 -- 172.21.15.27:0/682399215 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7fee44069840 con 0x7fee5012ca00 2024-09-21T03:28:50.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.095+0000 7fee4cff9640 1 -- 172.21.15.27:0/682399215 --> v1:172.21.15.27:6805/580113737 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee3c05baf0 con 0x7fee3c057df0 2024-09-21T03:28:50.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.095+0000 7fee4cff9640 1 -- 172.21.15.27:0/682399215 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fee4406e2c0 con 0x7fee5012ca00 2024-09-21T03:28:50.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.096+0000 7f2a46ffd640 1 -- 172.21.15.27:0/3285330373 <== osd.2 v1:172.21.15.27:6809/2016211836 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f2a3405bad0 con 0x7f2a34057dd0 2024-09-21T03:28:50.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.096+0000 7fee4cff9640 1 -- 172.21.15.27:0/682399215 <== osd.1 v1:172.21.15.27:6805/580113737 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fee3c05baf0 con 0x7fee3c057df0 2024-09-21T03:28:50.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.107+0000 7f006a7cb640 1 -- 172.21.15.27:0/2465637642 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f006412cf80 msgr2=0x7f00641294b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.107+0000 7f006a7cb640 1 --2- 172.21.15.27:0/2465637642 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f006412cf80 0x7f00641294b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f005c005fd0 tx=0x7f005c02aa00 comp rx=0 tx=0).stop 2024-09-21T03:28:50.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.107+0000 7f006a7cb640 1 -- 172.21.15.27:0/2465637642 shutdown_connections 2024-09-21T03:28:50.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.107+0000 7f006a7cb640 1 --2- 172.21.15.27:0/2465637642 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f006412cf80 0x7f00641294b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.107+0000 7f006a7cb640 1 --2- 172.21.15.27:0/2465637642 >> v2:172.21.15.27:3300/0 conn(0x7f00640727c0 0x7f006412c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.107+0000 7f006a7cb640 1 -- 172.21.15.27:0/2465637642 >> 172.21.15.27:0/2465637642 conn(0x7f006409eb10 msgr2=0x7f006409ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.107+0000 7f006a7cb640 1 -- 172.21.15.27:0/2465637642 shutdown_connections 2024-09-21T03:28:50.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.107+0000 7f006a7cb640 1 -- 172.21.15.27:0/2465637642 wait complete. 2024-09-21T03:28:50.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.108+0000 7f8ae451f640 1 -- 172.21.15.27:0/3551072271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8adc12fb20 msgr2=0x7f8adc12ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.108+0000 7f8ae451f640 1 --2- 172.21.15.27:0/3551072271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8adc12fb20 0x7f8adc12ff30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8009880 tx=0x7f8ad802a440 comp rx=0 tx=0).stop 2024-09-21T03:28:50.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.108+0000 7f8ae451f640 1 -- 172.21.15.27:0/3551072271 shutdown_connections 2024-09-21T03:28:50.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.108+0000 7f8ae451f640 1 --2- 172.21.15.27:0/3551072271 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8adc130470 0x7f8adc070850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.108+0000 7f8ae451f640 1 --2- 172.21.15.27:0/3551072271 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8adc12fb20 0x7f8adc12ff30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.108+0000 7f8ae451f640 1 -- 172.21.15.27:0/3551072271 >> 172.21.15.27:0/3551072271 conn(0x7f8adc09eb80 msgr2=0x7f8adc09ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.108+0000 7f8ae451f640 1 -- 172.21.15.27:0/3551072271 shutdown_connections 2024-09-21T03:28:50.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.108+0000 7f8ae451f640 1 -- 172.21.15.27:0/3551072271 wait complete. 2024-09-21T03:28:50.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.109+0000 7f006a7cb640 1 Processor -- start 2024-09-21T03:28:50.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.109+0000 7f8ae451f640 1 Processor -- start 2024-09-21T03:28:50.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.110+0000 7f8ae451f640 1 -- start start 2024-09-21T03:28:50.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.110+0000 7f006a7cb640 1 -- start start 2024-09-21T03:28:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.110+0000 7f8ae451f640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8adc070d90 0x7f8adc1c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.110+0000 7f8ae451f640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8adc12fb20 0x7f8adc1c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.110+0000 7f8ae451f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8adc130470 0x7f8adc1c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.110+0000 7f8ae22bb640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8adc070d90 0x7f8adc1c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.110+0000 7f8ae451f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8adc138590 con 0x7f8adc130470 2024-09-21T03:28:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.110+0000 7f8ae22bb640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8adc070d90 0x7f8adc1c59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40260/0 (socket says 172.21.15.27:40260) 2024-09-21T03:28:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.110+0000 7f8ae22bb640 1 -- 172.21.15.27:0/3101786854 learned_addr learned my addr 172.21.15.27:0/3101786854 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f8ae1aba640 1 --2- 172.21.15.27:0/3101786854 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8adc12fb20 0x7f8adc1c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f8ae2abc640 1 --2- 172.21.15.27:0/3101786854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8adc130470 0x7f8adc1c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f8adc138410 con 0x7f8adc070d90 2024-09-21T03:28:50.113 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f8adc138710 con 0x7f8adc12fb20 2024-09-21T03:28:50.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f006a7cb640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f00640727c0 0x7f0064096d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f006a7cb640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f006412cb70 0x7f0064097260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.114 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f8ae22bb640 1 -- 172.21.15.27:0/3101786854 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8adc12fb20 msgr2=0x7f8adc1c5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f0063fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f00640727c0 0x7f0064096d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f0063fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f00640727c0 0x7f0064096d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40266/0 (socket says 172.21.15.27:40266) 2024-09-21T03:28:50.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f0063fff640 1 -- 172.21.15.27:0/1358919128 learned_addr learned my addr 172.21.15.27:0/1358919128 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:50.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f8ae22bb640 1 --2- 172.21.15.27:0/3101786854 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8adc12fb20 0x7f8adc1c5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f8ae22bb640 1 -- 172.21.15.27:0/3101786854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8adc130470 msgr2=0x7f8adc1c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f8ae22bb640 1 --2- 172.21.15.27:0/3101786854 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8adc130470 0x7f8adc1c6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.115 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f8ae22bb640 1 -- 172.21.15.27:0/3101786854 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8adc0944c0 con 0x7f8adc070d90 2024-09-21T03:28:50.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f006a7cb640 1 --2- 172.21.15.27:0/1358919128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640977a0 0x7f00641c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f00641353a0 con 0x7f00640977a0 2024-09-21T03:28:50.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f8ae22bb640 1 --2- 172.21.15.27:0/3101786854 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8adc070d90 0x7f8adc1c59a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8000c00 tx=0x7f8ad8032c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:50.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.111+0000 7f00637fe640 1 --2- 172.21.15.27:0/1358919128 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f006412cb70 0x7f0064097260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.116 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f0064135220 con 0x7f00640727c0 2024-09-21T03:28:50.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f8ad2ffd640 1 -- 172.21.15.27:0/3101786854 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8040070 con 0x7f8adc070d90 2024-09-21T03:28:50.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f0064135520 con 0x7f006412cb70 2024-09-21T03:28:50.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f0068d68640 1 --2- 172.21.15.27:0/1358919128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640977a0 0x7f00641c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.117 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f0068d68640 1 -- 172.21.15.27:0/1358919128 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f006412cb70 msgr2=0x7f0064097260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f0068d68640 1 --2- 172.21.15.27:0/1358919128 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f006412cb70 0x7f0064097260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f8ad2ffd640 1 -- 172.21.15.27:0/3101786854 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8ad8003800 con 0x7f8adc070d90 2024-09-21T03:28:50.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f0068d68640 1 -- 172.21.15.27:0/1358919128 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f00640727c0 msgr2=0x7f0064096d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f0068d68640 1 --2- 172.21.15.27:0/1358919128 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f00640727c0 0x7f0064096d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f0068d68640 1 -- 172.21.15.27:0/1358919128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0064133f30 con 0x7f00640977a0 2024-09-21T03:28:50.118 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f0063fff640 1 --2- 172.21.15.27:0/1358919128 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f00640727c0 0x7f0064096d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:50.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.112+0000 7f8ad2ffd640 1 -- 172.21.15.27:0/3101786854 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8003bd0 con 0x7f8adc070d90 2024-09-21T03:28:50.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.113+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8adc095310 con 0x7f8adc070d90 2024-09-21T03:28:50.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.113+0000 7f0068d68640 1 --2- 172.21.15.27:0/1358919128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640977a0 0x7f00641c8820 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f005c032000 tx=0x7f005c0093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:50.119 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.113+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8adc092340 con 0x7f8adc070d90 2024-09-21T03:28:50.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.113+0000 7f0060ff9640 1 -- 172.21.15.27:0/1358919128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005c044020 con 0x7f00640977a0 2024-09-21T03:28:50.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.113+0000 7f0060ff9640 1 -- 172.21.15.27:0/1358919128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f005c004540 con 0x7f00640977a0 2024-09-21T03:28:50.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.113+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0064098550 con 0x7f00640977a0 2024-09-21T03:28:50.120 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.113+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8adc12ff30 con 0x7f8adc070d90 2024-09-21T03:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.114+0000 7f8ad2ffd640 1 -- 172.21.15.27:0/3101786854 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f8ad8039040 con 0x7f8adc070d90 2024-09-21T03:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.114+0000 7f0060ff9640 1 -- 172.21.15.27:0/1358919128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005c033070 con 0x7f00640977a0 2024-09-21T03:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.114+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f006412dd90 con 0x7f00640977a0 2024-09-21T03:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.115+0000 7f8ad2ffd640 1 -- 172.21.15.27:0/3101786854 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f8ad808a3e0 con 0x7f8adc070d90 2024-09-21T03:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.115+0000 7f8ad2ffd640 1 -- 172.21.15.27:0/3101786854 --> v1:172.21.15.27:6801/1914045614 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ad803be90 con 0x7f8ac0057df0 2024-09-21T03:28:50.121 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.115+0000 7f8ad2ffd640 1 -- 172.21.15.27:0/3101786854 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f8ad8091050 con 0x7f8adc070d90 2024-09-21T03:28:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.115+0000 7f0060ff9640 1 -- 172.21.15.27:0/1358919128 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f005c047020 con 0x7f00640977a0 2024-09-21T03:28:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.115+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f006412b150 con 0x7f00640977a0 2024-09-21T03:28:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.115+0000 7f0060ff9640 1 -- 172.21.15.27:0/1358919128 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f005c091db0 con 0x7f00640977a0 2024-09-21T03:28:50.122 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.116+0000 7f0060ff9640 1 -- 172.21.15.27:0/1358919128 --> v1:172.21.15.27:6813/1188450781 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f005c04ae50 con 0x7f0050057df0 2024-09-21T03:28:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.116+0000 7f0060ff9640 1 -- 172.21.15.27:0/1358919128 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f005c092060 con 0x7f00640977a0 2024-09-21T03:28:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.117+0000 7f0060ff9640 1 -- 172.21.15.27:0/1358919128 <== osd.3 v1:172.21.15.27:6813/1188450781 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f005c04ae50 con 0x7f0050057df0 2024-09-21T03:28:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.117+0000 7f8ad2ffd640 1 -- 172.21.15.27:0/3101786854 <== osd.0 v1:172.21.15.27:6801/1914045614 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8ad803be90 con 0x7f8ac0057df0 2024-09-21T03:28:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.118+0000 7f2a587f6640 1 -- 172.21.15.27:0/3285330373 --> v1:172.21.15.27:6809/2016211836 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2a50097490 con 0x7f2a34057dd0 2024-09-21T03:28:50.123 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.118+0000 7f2a46ffd640 1 -- 172.21.15.27:0/3285330373 <== osd.2 v1:172.21.15.27:6809/2016211836 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 277563570) 0x7f2a50097490 con 0x7f2a34057dd0 2024-09-21T03:28:50.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.119+0000 7f2a587f6640 1 -- 172.21.15.27:0/3285330373 >> v1:172.21.15.27:6809/2016211836 conn(0x7f2a34057dd0 legacy=0x7f2a3405a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.119+0000 7f2a587f6640 1 -- 172.21.15.27:0/3285330373 >> v1:172.21.15.27:6800/2405211194 conn(0x7f2a34051500 legacy=0x7f2a34053a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.119+0000 7f2a587f6640 1 -- 172.21.15.27:0/3285330373 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a5012d2f0 msgr2=0x7f2a501c3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.119+0000 7f2a587f6640 1 --2- 172.21.15.27:0/3285330373 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2a5012d2f0 0x7f2a501c3c50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2a500737b0 tx=0x7f2a40005d70 comp rx=0 tx=0).stop 2024-09-21T03:28:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.119+0000 7f2a56d93640 1 -- 172.21.15.27:0/3285330373 reap_dead start 2024-09-21T03:28:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.119+0000 7f2a587f6640 1 -- 172.21.15.27:0/3285330373 shutdown_connections 2024-09-21T03:28:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.119+0000 7f2a587f6640 1 -- 172.21.15.27:0/3285330373 >> 172.21.15.27:0/3285330373 conn(0x7f2a50098360 msgr2=0x7f2a501286e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.119+0000 7f2a587f6640 1 -- 172.21.15.27:0/3285330373 shutdown_connections 2024-09-21T03:28:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.119+0000 7f2a587f6640 1 -- 172.21.15.27:0/3285330373 wait complete. 2024-09-21T03:28:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.121+0000 7fee567cc640 1 -- 172.21.15.27:0/682399215 --> v1:172.21.15.27:6805/580113737 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fee50135660 con 0x7fee3c057df0 2024-09-21T03:28:50.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.122+0000 7fee4cff9640 1 -- 172.21.15.27:0/682399215 <== osd.1 v1:172.21.15.27:6805/580113737 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 3232526981) 0x7fee50135660 con 0x7fee3c057df0 2024-09-21T03:28:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.122+0000 7fee567cc640 1 -- 172.21.15.27:0/682399215 >> v1:172.21.15.27:6805/580113737 conn(0x7fee3c057df0 legacy=0x7fee3c05a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.122+0000 7fee567cc640 1 -- 172.21.15.27:0/682399215 >> v1:172.21.15.27:6800/2405211194 conn(0x7fee3c0514b0 legacy=0x7fee3c0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.122+0000 7fee567cc640 1 -- 172.21.15.27:0/682399215 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fee5012ca00 msgr2=0x7fee5009b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.122+0000 7fee567cc640 1 --2- 172.21.15.27:0/682399215 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fee5012ca00 0x7fee5009b9c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fee4400e9c0 tx=0x7fee4400ee80 comp rx=0 tx=0).stop 2024-09-21T03:28:50.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.122+0000 7fee54d69640 1 -- 172.21.15.27:0/682399215 reap_dead start 2024-09-21T03:28:50.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.122+0000 7fee567cc640 1 -- 172.21.15.27:0/682399215 shutdown_connections 2024-09-21T03:28:50.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.122+0000 7fee567cc640 1 -- 172.21.15.27:0/682399215 >> 172.21.15.27:0/682399215 conn(0x7fee500a1d70 msgr2=0x7fee5009c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.123+0000 7fee567cc640 1 -- 172.21.15.27:0/682399215 shutdown_connections 2024-09-21T03:28:50.127 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.123+0000 7fee567cc640 1 -- 172.21.15.27:0/682399215 wait complete. 2024-09-21T03:28:50.133 INFO:teuthology.orchestra.run.smithi027.stdout:124554051588 2024-09-21T03:28:50.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T03:28:50.136 INFO:teuthology.orchestra.run.smithi027.stdout:120259084292 2024-09-21T03:28:50.136 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T03:28:50.150 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.149+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 --> v1:172.21.15.27:6801/1914045614 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8adc0928b0 con 0x7f8ac0057df0 2024-09-21T03:28:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.149+0000 7f8ad2ffd640 1 -- 172.21.15.27:0/3101786854 <== osd.0 v1:172.21.15.27:6801/1914045614 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 3581622916) 0x7f8adc0928b0 con 0x7f8ac0057df0 2024-09-21T03:28:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.149+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 --> v1:172.21.15.27:6813/1188450781 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f006412cf80 con 0x7f0050057df0 2024-09-21T03:28:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.149+0000 7f0060ff9640 1 -- 172.21.15.27:0/1358919128 <== osd.3 v1:172.21.15.27:6813/1188450781 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (unknown 0 0 1985872850) 0x7f006412cf80 con 0x7f0050057df0 2024-09-21T03:28:50.151 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 >> v1:172.21.15.27:6801/1914045614 conn(0x7f8ac0057df0 legacy=0x7f8ac005a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 >> v1:172.21.15.27:6800/2405211194 conn(0x7f8ac00514b0 legacy=0x7f8ac00539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.152 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8adc070d90 msgr2=0x7f8adc1c59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f8ae451f640 1 --2- 172.21.15.27:0/3101786854 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8adc070d90 0x7f8adc1c59a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8000c00 tx=0x7f8ad8032c10 comp rx=0 tx=0).stop 2024-09-21T03:28:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f8ae2abc640 1 -- 172.21.15.27:0/3101786854 reap_dead start 2024-09-21T03:28:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 shutdown_connections 2024-09-21T03:28:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 >> 172.21.15.27:0/3101786854 conn(0x7f8adc09eb80 msgr2=0x7f8adc1370d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 shutdown_connections 2024-09-21T03:28:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f8ae451f640 1 -- 172.21.15.27:0/3101786854 wait complete. 2024-09-21T03:28:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 >> v1:172.21.15.27:6813/1188450781 conn(0x7f0050057df0 legacy=0x7f005005a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.153 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 >> v1:172.21.15.27:6800/2405211194 conn(0x7f00500514b0 legacy=0x7f00500539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640977a0 msgr2=0x7f00641c8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f006a7cb640 1 --2- 172.21.15.27:0/1358919128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640977a0 0x7f00641c8820 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f005c032000 tx=0x7f005c0093d0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f0068d68640 1 -- 172.21.15.27:0/1358919128 reap_dead start 2024-09-21T03:28:50.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 shutdown_connections 2024-09-21T03:28:50.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.150+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 >> 172.21.15.27:0/1358919128 conn(0x7f006409eb10 msgr2=0x7f00641239f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.151+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 shutdown_connections 2024-09-21T03:28:50.154 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.151+0000 7f006a7cb640 1 -- 172.21.15.27:0/1358919128 wait complete. 2024-09-21T03:28:50.165 INFO:teuthology.orchestra.run.smithi027.stdout:133143986179 2024-09-21T03:28:50.165 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T03:28:50.166 INFO:teuthology.orchestra.run.smithi027.stdout:115964116996 2024-09-21T03:28:50.166 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T03:28:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.255+0000 7f8096af6640 1 -- 172.21.15.27:0/891305907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8090130740 msgr2=0x7f8090132b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.255+0000 7f8096af6640 1 --2- 172.21.15.27:0/891305907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8090130740 0x7f8090132b60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f807c009830 tx=0x7f807c02a480 comp rx=0 tx=0).stop 2024-09-21T03:28:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.255+0000 7f8096af6640 1 -- 172.21.15.27:0/891305907 shutdown_connections 2024-09-21T03:28:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.255+0000 7f8096af6640 1 --2- 172.21.15.27:0/891305907 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f80901330a0 0x7f80901354c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.255+0000 7f8096af6640 1 --2- 172.21.15.27:0/891305907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8090130740 0x7f8090132b60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.255+0000 7f8096af6640 1 -- 172.21.15.27:0/891305907 >> 172.21.15.27:0/891305907 conn(0x7f809009eb80 msgr2=0x7f809009ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.255+0000 7f8096af6640 1 -- 172.21.15.27:0/891305907 shutdown_connections 2024-09-21T03:28:50.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.255+0000 7f8096af6640 1 -- 172.21.15.27:0/891305907 wait complete. 2024-09-21T03:28:50.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.256+0000 7f8096af6640 1 Processor -- start 2024-09-21T03:28:50.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.256+0000 7f8096af6640 1 -- start start 2024-09-21T03:28:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.257+0000 7f8096af6640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f809012a2e0 0x7f8090097960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.257+0000 7f8096af6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8090130740 0x7f8090097ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.257+0000 7f8096af6640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f80901330a0 0x7f80900983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.257+0000 7f8096af6640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f80901365c0 con 0x7f8090130740 2024-09-21T03:28:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.257+0000 7f8096af6640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f8090136440 con 0x7f80901330a0 2024-09-21T03:28:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.257+0000 7f8096af6640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f8090136740 con 0x7f809012a2e0 2024-09-21T03:28:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.257+0000 7f8095093640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f80901330a0 0x7f80900983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.257+0000 7f8095093640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f80901330a0 0x7f80900983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40282/0 (socket says 172.21.15.27:40282) 2024-09-21T03:28:50.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.257+0000 7f8095093640 1 -- 172.21.15.27:0/1120687272 learned_addr learned my addr 172.21.15.27:0/1120687272 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:50.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.257+0000 7f8094892640 1 --2- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f809012a2e0 0x7f8090097960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.260 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.258+0000 7f808ffff640 1 --2- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8090130740 0x7f8090097ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.258+0000 7f8095093640 1 -- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f809012a2e0 msgr2=0x7f8090097960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.258+0000 7f8095093640 1 --2- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f809012a2e0 0x7f8090097960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.258+0000 7f8095093640 1 -- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8090130740 msgr2=0x7f8090097ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.258+0000 7f8095093640 1 --2- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8090130740 0x7f8090097ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.258+0000 7f8095093640 1 -- 172.21.15.27:0/1120687272 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f809012f800 con 0x7f80901330a0 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.258+0000 7f8094892640 1 --2- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f809012a2e0 0x7f8090097960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.258+0000 7f808ffff640 1 --2- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8090130740 0x7f8090097ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.258+0000 7f8095093640 1 --2- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f80901330a0 0x7f80900983e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f808800ee70 tx=0x7f808800c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.259+0000 7f808d7fa640 1 -- 172.21.15.27:0/1120687272 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088004d20 con 0x7f80901330a0 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.259+0000 7f808d7fa640 1 -- 172.21.15.27:0/1120687272 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8088003970 con 0x7f80901330a0 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.259+0000 7f8096af6640 1 -- 172.21.15.27:0/1120687272 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8090070850 con 0x7f80901330a0 2024-09-21T03:28:50.261 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.259+0000 7f808d7fa640 1 -- 172.21.15.27:0/1120687272 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8088003dc0 con 0x7f80901330a0 2024-09-21T03:28:50.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.259+0000 7f8096af6640 1 -- 172.21.15.27:0/1120687272 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8090070d70 con 0x7f80901330a0 2024-09-21T03:28:50.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.260+0000 7f8096af6640 1 -- 172.21.15.27:0/1120687272 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8090072520 con 0x7f80901330a0 2024-09-21T03:28:50.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.261+0000 7f808d7fa640 1 -- 172.21.15.27:0/1120687272 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f808801d020 con 0x7f80901330a0 2024-09-21T03:28:50.262 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.261+0000 7f808d7fa640 1 -- 172.21.15.27:0/1120687272 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f8088069870 con 0x7f80901330a0 2024-09-21T03:28:50.270 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.268+0000 7f808d7fa640 1 -- 172.21.15.27:0/1120687272 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f8088068920 con 0x7f80901330a0 2024-09-21T03:28:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.273+0000 7f0fcc034640 1 -- 172.21.15.27:0/2516766913 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0fc4073c10 msgr2=0x7f0fc412a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.273+0000 7f0fcc034640 1 --2- 172.21.15.27:0/2516766913 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0fc4073c10 0x7f0fc412a390 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009900 tx=0x7f0fc002a440 comp rx=0 tx=0).stop 2024-09-21T03:28:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.273+0000 7f0fcc034640 1 -- 172.21.15.27:0/2516766913 shutdown_connections 2024-09-21T03:28:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.273+0000 7f0fcc034640 1 --2- 172.21.15.27:0/2516766913 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0fc412a8d0 0x7f0fc412d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.273+0000 7f0fcc034640 1 --2- 172.21.15.27:0/2516766913 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0fc4073c10 0x7f0fc412a390 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.275 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.273+0000 7f0fcc034640 1 -- 172.21.15.27:0/2516766913 >> 172.21.15.27:0/2516766913 conn(0x7f0fc409fa20 msgr2=0x7f0fc409fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.274+0000 7f0fcc034640 1 -- 172.21.15.27:0/2516766913 shutdown_connections 2024-09-21T03:28:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.274+0000 7f0fcc034640 1 -- 172.21.15.27:0/2516766913 wait complete. 2024-09-21T03:28:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.275+0000 7f0fcc034640 1 Processor -- start 2024-09-21T03:28:50.276 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.275+0000 7f0fcc034640 1 -- start start 2024-09-21T03:28:50.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.275+0000 7f0fcc034640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0fc4073c10 0x7f0fc4134ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.275+0000 7f0fcc034640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0fc412a8d0 0x7f0fc4135220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.275+0000 7f0fcc034640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc412ace0 0x7f0fc4097410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.275+0000 7f0fcc034640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f0fc4136670 con 0x7f0fc412ace0 2024-09-21T03:28:50.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.275+0000 7f0fcc034640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f0fc41364f0 con 0x7f0fc412a8d0 2024-09-21T03:28:50.277 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.275+0000 7f0fcc034640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f0fc41367f0 con 0x7f0fc4073c10 2024-09-21T03:28:50.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fc9dd0640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0fc4073c10 0x7f0fc4134ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fc9dd0640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0fc4073c10 0x7f0fc4134ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:50630/0 (socket says 172.21.15.27:50630) 2024-09-21T03:28:50.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fc9dd0640 1 -- 172.21.15.27:0/1403449775 learned_addr learned my addr 172.21.15.27:0/1403449775 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:50.279 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fc95cf640 1 --2- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0fc412a8d0 0x7f0fc4135220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fca5d1640 1 --2- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc412ace0 0x7f0fc4097410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fc95cf640 1 -- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0fc4073c10 msgr2=0x7f0fc4134ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fc95cf640 1 --2- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0fc4073c10 0x7f0fc4134ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fc95cf640 1 -- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc412ace0 msgr2=0x7f0fc4097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fc95cf640 1 --2- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc412ace0 0x7f0fc4097410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fc95cf640 1 -- 172.21.15.27:0/1403449775 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fc406fc10 con 0x7f0fc412a8d0 2024-09-21T03:28:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fca5d1640 1 --2- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc412ace0 0x7f0fc4097410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.276+0000 7f0fc95cf640 1 --2- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0fc412a8d0 0x7f0fc4135220 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0fb400e9f0 tx=0x7f0fb400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.277+0000 7f0fba7fc640 1 -- 172.21.15.27:0/1403449775 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb400ccc0 con 0x7f0fc412a8d0 2024-09-21T03:28:50.280 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.277+0000 7f0fba7fc640 1 -- 172.21.15.27:0/1403449775 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f0fb4004020 con 0x7f0fc412a8d0 2024-09-21T03:28:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.277+0000 7f0fcc034640 1 -- 172.21.15.27:0/1403449775 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fc4127d30 con 0x7f0fc412a8d0 2024-09-21T03:28:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.277+0000 7f0fba7fc640 1 -- 172.21.15.27:0/1403449775 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fb4004470 con 0x7f0fc412a8d0 2024-09-21T03:28:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.277+0000 7f0fcc034640 1 -- 172.21.15.27:0/1403449775 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fc40703a0 con 0x7f0fc412a8d0 2024-09-21T03:28:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.278+0000 7f0fcc034640 1 -- 172.21.15.27:0/1403449775 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fc412a390 con 0x7f0fc412a8d0 2024-09-21T03:28:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.279+0000 7f0fba7fc640 1 -- 172.21.15.27:0/1403449775 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f0fb4005020 con 0x7f0fc412a8d0 2024-09-21T03:28:50.281 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.279+0000 7f0fba7fc640 1 -- 172.21.15.27:0/1403449775 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f0fb4069730 con 0x7f0fc412a8d0 2024-09-21T03:28:50.288 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.287+0000 7f0fba7fc640 1 -- 172.21.15.27:0/1403449775 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f0fb40687e0 con 0x7f0fc412a8d0 2024-09-21T03:28:50.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7f75f89c0640 1 -- 172.21.15.27:0/1404410841 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f00047a0 con 0x7f75f412d570 2024-09-21T03:28:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7fd14a79d640 1 -- 172.21.15.27:0/4167988908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd144071700 msgr2=0x7fd1441292e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7fd14a79d640 1 --2- 172.21.15.27:0/4167988908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd144071700 0x7fd1441292e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd140009860 tx=0x7fd14002a410 comp rx=0 tx=0).stop 2024-09-21T03:28:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7fd14a79d640 1 -- 172.21.15.27:0/4167988908 shutdown_connections 2024-09-21T03:28:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7fd14a79d640 1 --2- 172.21.15.27:0/4167988908 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd144129820 0x7fd14412c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7fd14a79d640 1 --2- 172.21.15.27:0/4167988908 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd144128db0 0x7fd14412c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7fd14a79d640 1 --2- 172.21.15.27:0/4167988908 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd144071700 0x7fd1441292e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7fd14a79d640 1 -- 172.21.15.27:0/4167988908 >> 172.21.15.27:0/4167988908 conn(0x7fd14409e720 msgr2=0x7fd14409eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7f75fbc26640 1 -- 172.21.15.27:0/1404410841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f412d570 msgr2=0x7f75f412d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7f75fbc26640 1 --2- 172.21.15.27:0/1404410841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f412d570 0x7f75f412d980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f75f0005fd0 tx=0x7f75f002a420 comp rx=0 tx=0).stop 2024-09-21T03:28:50.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7fd14a79d640 1 -- 172.21.15.27:0/4167988908 shutdown_connections 2024-09-21T03:28:50.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7fd14a79d640 1 -- 172.21.15.27:0/4167988908 wait complete. 2024-09-21T03:28:50.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7f75fbc26640 1 -- 172.21.15.27:0/1404410841 shutdown_connections 2024-09-21T03:28:50.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7f75fbc26640 1 --2- 172.21.15.27:0/1404410841 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f75f4094580 0x7f75f4094990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7f75fbc26640 1 --2- 172.21.15.27:0/1404410841 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f75f412dec0 0x7f75f4094040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7f75fbc26640 1 --2- 172.21.15.27:0/1404410841 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f412d570 0x7f75f412d980 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7f75fbc26640 1 -- 172.21.15.27:0/1404410841 >> 172.21.15.27:0/1404410841 conn(0x7f75f409e5e0 msgr2=0x7f75f409aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7f75fbc26640 1 -- 172.21.15.27:0/1404410841 shutdown_connections 2024-09-21T03:28:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.333+0000 7f75fbc26640 1 -- 172.21.15.27:0/1404410841 wait complete. 2024-09-21T03:28:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75fbc26640 1 Processor -- start 2024-09-21T03:28:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75fbc26640 1 -- start start 2024-09-21T03:28:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75fbc26640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f75f4094580 0x7f75f4126e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75fbc26640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f412d570 0x7f75f41273c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75fbc26640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f75f412dec0 0x7f75f4127900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75fbc26640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f75f4070710 con 0x7f75f412d570 2024-09-21T03:28:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75fbc26640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f75f4070590 con 0x7f75f4094580 2024-09-21T03:28:50.338 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75fbc26640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f75f4070890 con 0x7f75f412dec0 2024-09-21T03:28:50.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7fd14a79d640 1 Processor -- start 2024-09-21T03:28:50.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75f99c2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f75f4094580 0x7f75f4126e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75f99c2640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f75f4094580 0x7f75f4126e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40332/0 (socket says 172.21.15.27:40332) 2024-09-21T03:28:50.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7fd14a79d640 1 -- start start 2024-09-21T03:28:50.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75f99c2640 1 -- 172.21.15.27:0/3214336221 learned_addr learned my addr 172.21.15.27:0/3214336221 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:50.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75f91c1640 1 --2- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f412d570 0x7f75f41273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.335+0000 7f75fa1c3640 1 --2- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f75f412dec0 0x7f75f4127900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75f99c2640 1 -- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f75f412dec0 msgr2=0x7f75f4127900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75f99c2640 1 --2- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f75f412dec0 0x7f75f4127900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75f99c2640 1 -- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f412d570 msgr2=0x7f75f41273c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75f99c2640 1 --2- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f412d570 0x7f75f41273c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75f99c2640 1 -- 172.21.15.27:0/3214336221 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f41290f0 con 0x7f75f4094580 2024-09-21T03:28:50.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75f91c1640 1 --2- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f412d570 0x7f75f41273c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:50.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75fa1c3640 1 --2- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f75f412dec0 0x7f75f4127900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:50.341 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75f99c2640 1 --2- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f75f4094580 0x7f75f4126e80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009850 tx=0x7f75f0005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:50.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75ea7fc640 1 -- 172.21.15.27:0/3214336221 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f003f070 con 0x7f75f4094580 2024-09-21T03:28:50.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75ea7fc640 1 -- 172.21.15.27:0/3214336221 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f75f00047a0 con 0x7f75f4094580 2024-09-21T03:28:50.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75ea7fc640 1 -- 172.21.15.27:0/3214336221 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0035030 con 0x7f75f4094580 2024-09-21T03:28:50.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.336+0000 7f75fbc26640 1 -- 172.21.15.27:0/3214336221 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f4129ef0 con 0x7f75f4094580 2024-09-21T03:28:50.342 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.337+0000 7f75fbc26640 1 -- 172.21.15.27:0/3214336221 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f41c7900 con 0x7f75f4094580 2024-09-21T03:28:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.337+0000 7fd14a79d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd144071700 0x7fd14412f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.337+0000 7fd14a79d640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd144128db0 0x7fd14412f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.337+0000 7fd14979b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd144071700 0x7fd14412f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.337+0000 7fd14979b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd144071700 0x7fd14412f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40336/0 (socket says 172.21.15.27:40336) 2024-09-21T03:28:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.337+0000 7fd14979b640 1 -- 172.21.15.27:0/1529531096 learned_addr learned my addr 172.21.15.27:0/1529531096 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.338+0000 7fd14a79d640 1 --2- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd144129820 0x7fd14412feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.338+0000 7fd148f9a640 1 --2- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd144128db0 0x7fd14412f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.343 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.338+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd14406f4e0 con 0x7fd144129820 2024-09-21T03:28:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.338+0000 7f75fbc26640 1 -- 172.21.15.27:0/3214336221 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f412d980 con 0x7f75f4094580 2024-09-21T03:28:50.344 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.338+0000 7fd149f9c640 1 --2- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd144129820 0x7fd14412feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.338+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fd14406f360 con 0x7fd144071700 2024-09-21T03:28:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.338+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fd14406f660 con 0x7fd144128db0 2024-09-21T03:28:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.339+0000 7fd149f9c640 1 -- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd144128db0 msgr2=0x7fd14412f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.339+0000 7fd149f9c640 1 --2- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd144128db0 0x7fd14412f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.339+0000 7fd149f9c640 1 -- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd144071700 msgr2=0x7fd14412f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.339+0000 7fd149f9c640 1 --2- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd144071700 0x7fd14412f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.338+0000 7f75ea7fc640 1 -- 172.21.15.27:0/3214336221 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f75f0042070 con 0x7f75f4094580 2024-09-21T03:28:50.345 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.339+0000 7f75ea7fc640 1 -- 172.21.15.27:0/3214336221 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f75f0089d10 con 0x7f75f4094580 2024-09-21T03:28:50.346 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.339+0000 7fd149f9c640 1 -- 172.21.15.27:0/1529531096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd144132a90 con 0x7fd144129820 2024-09-21T03:28:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.339+0000 7fd149f9c640 1 --2- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd144129820 0x7fd14412feb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd13c005fd0 tx=0x7fd13c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.339+0000 7fd14979b640 1 --2- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd144071700 0x7fd14412f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.340+0000 7fd139ffb640 1 -- 172.21.15.27:0/1529531096 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c003970 con 0x7fd144129820 2024-09-21T03:28:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.340+0000 7fd139ffb640 1 -- 172.21.15.27:0/1529531096 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd13c003b10 con 0x7fd144129820 2024-09-21T03:28:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.340+0000 7fd139ffb640 1 -- 172.21.15.27:0/1529531096 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13c004020 con 0x7fd144129820 2024-09-21T03:28:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.341+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd144131780 con 0x7fd144129820 2024-09-21T03:28:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.341+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd144098080 con 0x7fd144129820 2024-09-21T03:28:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.343+0000 7fd139ffb640 1 -- 172.21.15.27:0/1529531096 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7fd13c0041c0 con 0x7fd144129820 2024-09-21T03:28:50.347 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.343+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1441292e0 con 0x7fd144129820 2024-09-21T03:28:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.343+0000 7f75ea7fc640 1 -- 172.21.15.27:0/3214336221 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f75f0088d20 con 0x7f75f4094580 2024-09-21T03:28:50.348 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.343+0000 7fd139ffb640 1 -- 172.21.15.27:0/1529531096 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7fd13c0683e0 con 0x7fd144129820 2024-09-21T03:28:50.352 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.351+0000 7fd139ffb640 1 -- 172.21.15.27:0/1529531096 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fd13c067f20 con 0x7fd144129820 2024-09-21T03:28:50.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.443+0000 7f8096af6640 1 -- 172.21.15.27:0/1120687272 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f809012a6f0 con 0x7f80901330a0 2024-09-21T03:28:50.444 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.443+0000 7f808d7fa640 1 -- 172.21.15.27:0/1120687272 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8088048090 con 0x7f80901330a0 2024-09-21T03:28:50.445 INFO:teuthology.orchestra.run.smithi027.stdout:124554051586 2024-09-21T03:28:50.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.445+0000 7f8096af6640 1 -- 172.21.15.27:0/1120687272 >> v1:172.21.15.27:6800/2405211194 conn(0x7f80740514b0 legacy=0x7f80740539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.446 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.445+0000 7f8096af6640 1 -- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f80901330a0 msgr2=0x7f80900983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.445+0000 7f8096af6640 1 --2- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f80901330a0 0x7f80900983e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f808800ee70 tx=0x7f808800c740 comp rx=0 tx=0).stop 2024-09-21T03:28:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.445+0000 7f8096af6640 1 -- 172.21.15.27:0/1120687272 shutdown_connections 2024-09-21T03:28:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.445+0000 7f8096af6640 1 --2- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f80901330a0 0x7f80900983e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.445+0000 7f8096af6640 1 --2- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8090130740 0x7f8090097ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.445+0000 7f8096af6640 1 --2- 172.21.15.27:0/1120687272 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f809012a2e0 0x7f8090097960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.445+0000 7f8096af6640 1 -- 172.21.15.27:0/1120687272 >> 172.21.15.27:0/1120687272 conn(0x7f809009eb80 msgr2=0x7f8090128c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.445+0000 7f8096af6640 1 -- 172.21.15.27:0/1120687272 shutdown_connections 2024-09-21T03:28:50.447 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.445+0000 7f8096af6640 1 -- 172.21.15.27:0/1120687272 wait complete. 2024-09-21T03:28:50.458 INFO:tasks.ceph.ceph_manager.ceph:need seq 124554051588 got 124554051586 for osd.2 2024-09-21T03:28:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.462+0000 7f0fcc034640 1 -- 172.21.15.27:0/1403449775 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0fc40987d0 con 0x7f0fc412a8d0 2024-09-21T03:28:50.463 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.462+0000 7f0fba7fc640 1 -- 172.21.15.27:0/1403449775 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0fb4068320 con 0x7f0fc412a8d0 2024-09-21T03:28:50.464 INFO:teuthology.orchestra.run.smithi027.stdout:120259084290 2024-09-21T03:28:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.464+0000 7f0fcc034640 1 -- 172.21.15.27:0/1403449775 >> v1:172.21.15.27:6800/2405211194 conn(0x7f0fa80514b0 legacy=0x7f0fa80539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.464+0000 7f0fcc034640 1 -- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0fc412a8d0 msgr2=0x7f0fc4135220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.464+0000 7f0fcc034640 1 --2- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0fc412a8d0 0x7f0fc4135220 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0fb400e9f0 tx=0x7f0fb400eeb0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.465+0000 7f0fcc034640 1 -- 172.21.15.27:0/1403449775 shutdown_connections 2024-09-21T03:28:50.466 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.465+0000 7f0fcc034640 1 --2- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0fc412ace0 0x7f0fc4097410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.465+0000 7f0fcc034640 1 --2- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f0fc412a8d0 0x7f0fc4135220 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.465+0000 7f0fcc034640 1 --2- 172.21.15.27:0/1403449775 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f0fc4073c10 0x7f0fc4134ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.465+0000 7f0fcc034640 1 -- 172.21.15.27:0/1403449775 >> 172.21.15.27:0/1403449775 conn(0x7f0fc409fa20 msgr2=0x7f0fc4099760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.465+0000 7f0fcc034640 1 -- 172.21.15.27:0/1403449775 shutdown_connections 2024-09-21T03:28:50.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.465+0000 7f0fcc034640 1 -- 172.21.15.27:0/1403449775 wait complete. 2024-09-21T03:28:50.478 INFO:tasks.ceph.ceph_manager.ceph:need seq 120259084292 got 120259084290 for osd.1 2024-09-21T03:28:50.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.505+0000 7f75fbc26640 1 -- 172.21.15.27:0/3214336221 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f75f40744a0 con 0x7f75f4094580 2024-09-21T03:28:50.507 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.506+0000 7f75ea7fc640 1 -- 172.21.15.27:0/3214336221 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f75f00888e0 con 0x7f75f4094580 2024-09-21T03:28:50.507 INFO:teuthology.orchestra.run.smithi027.stdout:115964116995 2024-09-21T03:28:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.508+0000 7f75fbc26640 1 -- 172.21.15.27:0/3214336221 >> v1:172.21.15.27:6800/2405211194 conn(0x7f75d8051580 legacy=0x7f75d8053a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.508+0000 7f75fbc26640 1 -- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f75f4094580 msgr2=0x7f75f4126e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.508+0000 7f75fbc26640 1 --2- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f75f4094580 0x7f75f4126e80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f75f0009850 tx=0x7f75f0005e80 comp rx=0 tx=0).stop 2024-09-21T03:28:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.508+0000 7f75fbc26640 1 -- 172.21.15.27:0/3214336221 shutdown_connections 2024-09-21T03:28:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.508+0000 7f75fbc26640 1 --2- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f75f412dec0 0x7f75f4127900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.508+0000 7f75fbc26640 1 --2- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f75f412d570 0x7f75f41273c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.508+0000 7f75fbc26640 1 --2- 172.21.15.27:0/3214336221 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f75f4094580 0x7f75f4126e80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.508+0000 7f75fbc26640 1 -- 172.21.15.27:0/3214336221 >> 172.21.15.27:0/3214336221 conn(0x7f75f409e5e0 msgr2=0x7f75f409a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.508+0000 7f75fbc26640 1 -- 172.21.15.27:0/3214336221 shutdown_connections 2024-09-21T03:28:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.508+0000 7f75fbc26640 1 -- 172.21.15.27:0/3214336221 wait complete. 2024-09-21T03:28:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.514+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd144131ed0 con 0x7fd144129820 2024-09-21T03:28:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.515+0000 7fd139ffb640 1 -- 172.21.15.27:0/1529531096 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd13c041ef0 con 0x7fd144129820 2024-09-21T03:28:50.516 INFO:teuthology.orchestra.run.smithi027.stdout:133143986178 2024-09-21T03:28:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.517+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 >> v1:172.21.15.27:6800/2405211194 conn(0x7fd1280514b0 legacy=0x7fd1280539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.517+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd144129820 msgr2=0x7fd14412feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.517+0000 7fd14a79d640 1 --2- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd144129820 0x7fd14412feb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd13c005fd0 tx=0x7fd13c00c2d0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.517+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 shutdown_connections 2024-09-21T03:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.517+0000 7fd14a79d640 1 --2- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd144129820 0x7fd14412feb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.517+0000 7fd14a79d640 1 --2- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd144128db0 0x7fd14412f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.517+0000 7fd14a79d640 1 --2- 172.21.15.27:0/1529531096 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd144071700 0x7fd14412f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.517+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 >> 172.21.15.27:0/1529531096 conn(0x7fd14409e720 msgr2=0x7fd144127520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.517+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 shutdown_connections 2024-09-21T03:28:50.519 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:50.517+0000 7fd14a79d640 1 -- 172.21.15.27:0/1529531096 wait complete. 2024-09-21T03:28:50.521 INFO:tasks.ceph.ceph_manager.ceph:need seq 115964116996 got 115964116995 for osd.0 2024-09-21T03:28:50.529 INFO:tasks.ceph.ceph_manager.ceph:need seq 133143986179 got 133143986178 for osd.3 2024-09-21T03:28:51.459 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T03:28:51.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T03:28:51.521 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T03:28:51.530 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T03:28:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.594+0000 7f09f591b640 1 -- 172.21.15.27:0/2660301290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f012d570 msgr2=0x7f09f012d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.594+0000 7f09f591b640 1 --2- 172.21.15.27:0/2660301290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f012d570 0x7f09f012d980 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f09e0009830 tx=0x7f09e002a3f0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.594+0000 7f09f591b640 1 -- 172.21.15.27:0/2660301290 shutdown_connections 2024-09-21T03:28:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.594+0000 7f09f591b640 1 --2- 172.21.15.27:0/2660301290 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f09f012dec0 0x7f09f0094040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.594+0000 7f09f591b640 1 --2- 172.21.15.27:0/2660301290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f012d570 0x7f09f012d980 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.594+0000 7f09f591b640 1 -- 172.21.15.27:0/2660301290 >> 172.21.15.27:0/2660301290 conn(0x7f09f009e5e0 msgr2=0x7f09f009aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:51.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.594+0000 7f09f591b640 1 -- 172.21.15.27:0/2660301290 shutdown_connections 2024-09-21T03:28:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.594+0000 7f09f591b640 1 -- 172.21.15.27:0/2660301290 wait complete. 2024-09-21T03:28:51.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.596+0000 7f09f591b640 1 Processor -- start 2024-09-21T03:28:51.598 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.597+0000 7f09f591b640 1 -- start start 2024-09-21T03:28:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09f591b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f09f0094580 0x7f09f0133ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09f591b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f012d570 0x7f09f0134410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09f591b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f09f012dec0 0x7f09f0134950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09f591b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f09f0136550 con 0x7f09f012d570 2024-09-21T03:28:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09f591b640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f09f01363d0 con 0x7f09f0094580 2024-09-21T03:28:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09f591b640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f09f01366d0 con 0x7f09f012dec0 2024-09-21T03:28:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09ee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f012d570 0x7f09f0134410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09ef7fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f09f012dec0 0x7f09f0134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09ee7fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f012d570 0x7f09f0134410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38050/0 (socket says 172.21.15.27:38050) 2024-09-21T03:28:51.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09ee7fc640 1 -- 172.21.15.27:0/117368533 learned_addr learned my addr 172.21.15.27:0/117368533 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:51.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09eeffd640 1 --2- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f09f0094580 0x7f09f0133ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09ee7fc640 1 -- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f09f012dec0 msgr2=0x7f09f0134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09ee7fc640 1 --2- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f09f012dec0 0x7f09f0134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09ee7fc640 1 -- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f09f0094580 msgr2=0x7f09f0133ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.603 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09ee7fc640 1 --2- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f09f0094580 0x7f09f0133ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09ee7fc640 1 -- 172.21.15.27:0/117368533 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f01234b0 con 0x7f09f012d570 2024-09-21T03:28:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.598+0000 7f09ef7fe640 1 --2- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f09f012dec0 0x7f09f0134950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.599+0000 7f09eeffd640 1 --2- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f09f0094580 0x7f09f0133ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.599+0000 7f09ee7fc640 1 --2- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f012d570 0x7f09f0134410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f09e400e790 tx=0x7f09e400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.599+0000 7f09d7fff640 1 -- 172.21.15.27:0/117368533 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e400eea0 con 0x7f09f012d570 2024-09-21T03:28:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.599+0000 7f09d7fff640 1 -- 172.21.15.27:0/117368533 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f09e400cc60 con 0x7f09f012d570 2024-09-21T03:28:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.599+0000 7f09d7fff640 1 -- 172.21.15.27:0/117368533 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e4005280 con 0x7f09f012d570 2024-09-21T03:28:51.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.599+0000 7f09f591b640 1 -- 172.21.15.27:0/117368533 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f0123890 con 0x7f09f012d570 2024-09-21T03:28:51.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.599+0000 7f09f591b640 1 -- 172.21.15.27:0/117368533 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f01c99d0 con 0x7f09f012d570 2024-09-21T03:28:51.605 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.600+0000 7f09f591b640 1 -- 172.21.15.27:0/117368533 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f012d980 con 0x7f09f012d570 2024-09-21T03:28:51.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.601+0000 7f09d7fff640 1 -- 172.21.15.27:0/117368533 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f09e4005420 con 0x7f09f012d570 2024-09-21T03:28:51.606 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.602+0000 7f09d7fff640 1 -- 172.21.15.27:0/117368533 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f09e40698a0 con 0x7f09f012d570 2024-09-21T03:28:51.607 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.606+0000 7f09d7fff640 1 -- 172.21.15.27:0/117368533 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f09e4068950 con 0x7f09f012d570 2024-09-21T03:28:51.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.638+0000 7ff2dee6f640 1 -- 172.21.15.27:0/3339992561 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff2d8072980 msgr2=0x7ff2d81291b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.638+0000 7ff2dee6f640 1 --2- 172.21.15.27:0/3339992561 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff2d8072980 0x7ff2d81291b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009830 tx=0x7ff2cc02a480 comp rx=0 tx=0).stop 2024-09-21T03:28:51.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.638+0000 7ff2dee6f640 1 -- 172.21.15.27:0/3339992561 shutdown_connections 2024-09-21T03:28:51.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.638+0000 7ff2dee6f640 1 --2- 172.21.15.27:0/3339992561 >> v2:172.21.15.27:3300/0 conn(0x7ff2d81296f0 0x7ff2d812c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.638+0000 7ff2dee6f640 1 --2- 172.21.15.27:0/3339992561 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff2d8072980 0x7ff2d81291b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.638+0000 7ff2dee6f640 1 -- 172.21.15.27:0/3339992561 >> 172.21.15.27:0/3339992561 conn(0x7ff2d809edc0 msgr2=0x7ff2d809f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:51.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.638+0000 7ff2dee6f640 1 -- 172.21.15.27:0/3339992561 shutdown_connections 2024-09-21T03:28:51.640 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.638+0000 7ff2dee6f640 1 -- 172.21.15.27:0/3339992561 wait complete. 2024-09-21T03:28:51.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.639+0000 7ff2dee6f640 1 Processor -- start 2024-09-21T03:28:51.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.639+0000 7ff2dee6f640 1 -- start start 2024-09-21T03:28:51.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.640+0000 7ff2dee6f640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff2d8072980 0x7ff2d8098080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.640+0000 7ff2dee6f640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff2d81296f0 0x7ff2d80985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.643 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.640+0000 7ff2dee6f640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d812c8f0 0x7ff2d8098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.640+0000 7ff2dee6f640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d81353b0 con 0x7ff2d812c8f0 2024-09-21T03:28:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.640+0000 7ff2dee6f640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7ff2d8135230 con 0x7ff2d81296f0 2024-09-21T03:28:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.640+0000 7ff2dee6f640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7ff2d8135530 con 0x7ff2d8072980 2024-09-21T03:28:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2dcc0b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff2d8072980 0x7ff2d8098080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2d7fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff2d81296f0 0x7ff2d80985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2d7fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff2d81296f0 0x7ff2d80985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40368/0 (socket says 172.21.15.27:40368) 2024-09-21T03:28:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2dcc0b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff2d8072980 0x7ff2d8098080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:50690/0 (socket says 172.21.15.27:50690) 2024-09-21T03:28:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2d7fff640 1 -- 172.21.15.27:0/345740323 learned_addr learned my addr 172.21.15.27:0/345740323 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2dd40c640 1 --2- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d812c8f0 0x7ff2d8098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2d7fff640 1 -- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff2d8072980 msgr2=0x7ff2d8098080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.644 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2d7fff640 1 --2- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff2d8072980 0x7ff2d8098080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2d7fff640 1 -- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d812c8f0 msgr2=0x7ff2d8098b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2d7fff640 1 --2- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d812c8f0 0x7ff2d8098b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2d7fff640 1 -- 172.21.15.27:0/345740323 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d81325a0 con 0x7ff2d81296f0 2024-09-21T03:28:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2dd40c640 1 --2- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d812c8f0 0x7ff2d8098b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2dcc0b640 1 --2- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff2d8072980 0x7ff2d8098080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.641+0000 7ff2d7fff640 1 --2- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff2d81296f0 0x7ff2d80985c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400e9b0 tx=0x7ff2c400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.642+0000 7ff2d57fa640 1 -- 172.21.15.27:0/345740323 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c400cd80 con 0x7ff2d81296f0 2024-09-21T03:28:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.642+0000 7ff2dee6f640 1 -- 172.21.15.27:0/345740323 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d806ecb0 con 0x7ff2d81296f0 2024-09-21T03:28:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.642+0000 7ff2d57fa640 1 -- 172.21.15.27:0/345740323 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ff2c4010430 con 0x7ff2d81296f0 2024-09-21T03:28:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.642+0000 7ff2d57fa640 1 -- 172.21.15.27:0/345740323 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4010860 con 0x7ff2d81296f0 2024-09-21T03:28:51.645 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.642+0000 7ff2dee6f640 1 -- 172.21.15.27:0/345740323 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d81314b0 con 0x7ff2d81296f0 2024-09-21T03:28:51.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.643+0000 7ff2dee6f640 1 -- 172.21.15.27:0/345740323 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d81291b0 con 0x7ff2d81296f0 2024-09-21T03:28:51.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.643+0000 7ff2d57fa640 1 -- 172.21.15.27:0/345740323 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7ff2c401f020 con 0x7ff2d81296f0 2024-09-21T03:28:51.646 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.644+0000 7ff2d57fa640 1 -- 172.21.15.27:0/345740323 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7ff2c4069820 con 0x7ff2d81296f0 2024-09-21T03:28:51.649 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.648+0000 7ff2d57fa640 1 -- 172.21.15.27:0/345740323 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7ff2c40688d0 con 0x7ff2d81296f0 2024-09-21T03:28:51.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.675+0000 7fde2dc68640 1 -- 172.21.15.27:0/1559025253 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fde2812c4c0 msgr2=0x7fde28071860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.675+0000 7fde2dc68640 1 --2- 172.21.15.27:0/1559025253 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fde2812c4c0 0x7fde28071860 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fde18009930 tx=0x7fde1802a560 comp rx=0 tx=0).stop 2024-09-21T03:28:51.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.675+0000 7f08d4813640 1 -- 172.21.15.27:0/1371314248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d012c4c0 msgr2=0x7f08d0071860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.675+0000 7f08d4813640 1 --2- 172.21.15.27:0/1371314248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d012c4c0 0x7f08d0071860 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f08c000d930 tx=0x7f08c002a160 comp rx=0 tx=0).stop 2024-09-21T03:28:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7fde2dc68640 1 -- 172.21.15.27:0/1559025253 shutdown_connections 2024-09-21T03:28:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7fde2dc68640 1 --2- 172.21.15.27:0/1559025253 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fde2812c9f0 0x7fde2812ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7fde2dc68640 1 --2- 172.21.15.27:0/1559025253 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fde2812c4c0 0x7fde28071860 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7fde2dc68640 1 -- 172.21.15.27:0/1559025253 >> 172.21.15.27:0/1559025253 conn(0x7fde2809eb80 msgr2=0x7fde2809ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7fde2dc68640 1 -- 172.21.15.27:0/1559025253 shutdown_connections 2024-09-21T03:28:51.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7fde2dc68640 1 -- 172.21.15.27:0/1559025253 wait complete. 2024-09-21T03:28:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7f08d4813640 1 -- 172.21.15.27:0/1371314248 shutdown_connections 2024-09-21T03:28:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7f08d4813640 1 --2- 172.21.15.27:0/1371314248 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f08d0128e10 0x7f08d0129220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7f08d4813640 1 --2- 172.21.15.27:0/1371314248 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f08d012c9f0 0x7f08d012ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7f08d4813640 1 --2- 172.21.15.27:0/1371314248 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d012c4c0 0x7f08d0071860 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7f08d4813640 1 -- 172.21.15.27:0/1371314248 >> 172.21.15.27:0/1371314248 conn(0x7f08d009eb80 msgr2=0x7f08d009ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7f08d4813640 1 -- 172.21.15.27:0/1371314248 shutdown_connections 2024-09-21T03:28:51.679 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.676+0000 7f08d4813640 1 -- 172.21.15.27:0/1371314248 wait complete. 2024-09-21T03:28:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.677+0000 7fde2dc68640 1 Processor -- start 2024-09-21T03:28:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.677+0000 7f08d4813640 1 Processor -- start 2024-09-21T03:28:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.677+0000 7fde2dc68640 1 -- start start 2024-09-21T03:28:51.680 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.677+0000 7f08d4813640 1 -- start start 2024-09-21T03:28:51.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde2dc68640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fde28128e10 0x7fde28096f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde2dc68640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fde2812c4c0 0x7fde28097450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde2dc68640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2812c9f0 0x7fde28097990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde2dc68640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fde2806f640 con 0x7fde2812c9f0 2024-09-21T03:28:51.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde2dc68640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fde2806f4c0 con 0x7fde2812c4c0 2024-09-21T03:28:51.681 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08d4813640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d0128e10 0x7f08d01c12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde2dc68640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fde2806f7c0 con 0x7fde28128e10 2024-09-21T03:28:51.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08d4813640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f08d012c4c0 0x7f08d01c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.682 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde277fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fde28128e10 0x7fde28096f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08ce59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d0128e10 0x7f08d01c12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde277fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fde28128e10 0x7fde28096f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:50718/0 (socket says 172.21.15.27:50718) 2024-09-21T03:28:51.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08cdd9b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f08d012c4c0 0x7f08d01c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.683 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08cdd9b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f08d012c4c0 0x7f08d01c17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40410/0 (socket says 172.21.15.27:40410) 2024-09-21T03:28:51.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08ce59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d0128e10 0x7f08d01c12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38086/0 (socket says 172.21.15.27:38086) 2024-09-21T03:28:51.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08cdd9b640 1 -- 172.21.15.27:0/838016413 learned_addr learned my addr 172.21.15.27:0/838016413 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:51.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08d4813640 1 --2- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f08d012c9f0 0x7f08d01c1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:51.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f08d006f430 con 0x7f08d0128e10 2024-09-21T03:28:51.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f08d006f2b0 con 0x7f08d012c4c0 2024-09-21T03:28:51.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f08d006f5b0 con 0x7f08d012c9f0 2024-09-21T03:28:51.684 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde277fe640 1 -- 172.21.15.27:0/3871505633 learned_addr learned my addr 172.21.15.27:0/3871505633 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:51.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde26ffd640 1 --2- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fde2812c4c0 0x7fde28097450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde27fff640 1 --2- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2812c9f0 0x7fde28097990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde26ffd640 1 -- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fde28128e10 msgr2=0x7fde28096f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde26ffd640 1 --2- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fde28128e10 0x7fde28096f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde26ffd640 1 -- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2812c9f0 msgr2=0x7fde28097990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde26ffd640 1 --2- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2812c9f0 0x7fde28097990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde26ffd640 1 -- 172.21.15.27:0/3871505633 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2812d590 con 0x7fde2812c4c0 2024-09-21T03:28:51.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7fde27fff640 1 --2- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2812c9f0 0x7fde28097990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:51.685 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.678+0000 7f08ced9d640 1 --2- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f08d012c9f0 0x7f08d01c1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:51.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7fde26ffd640 1 --2- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fde2812c4c0 0x7fde28097450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00e6d0 tx=0x7fde1c00eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:51.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7f08ce59c640 1 -- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f08d012c9f0 msgr2=0x7f08d01c1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7f08ce59c640 1 --2- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f08d012c9f0 0x7f08d01c1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7f08ce59c640 1 -- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f08d012c4c0 msgr2=0x7f08d01c17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.686 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7fde0ffff640 1 -- 172.21.15.27:0/3871505633 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c00cab0 con 0x7fde2812c4c0 2024-09-21T03:28:51.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7f08ce59c640 1 --2- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f08d012c4c0 0x7f08d01c17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7f08ce59c640 1 -- 172.21.15.27:0/838016413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08d0090500 con 0x7f08d0128e10 2024-09-21T03:28:51.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7fde0ffff640 1 -- 172.21.15.27:0/3871505633 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fde1c00cd40 con 0x7fde2812c4c0 2024-09-21T03:28:51.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7fde2dc68640 1 -- 172.21.15.27:0/3871505633 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde2812e090 con 0x7fde2812c4c0 2024-09-21T03:28:51.687 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7fde0ffff640 1 -- 172.21.15.27:0/3871505633 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1c010690 con 0x7fde2812c4c0 2024-09-21T03:28:51.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7f08ce59c640 1 --2- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d0128e10 0x7f08d01c12b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f08d0072300 tx=0x7f08c0003b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:51.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.679+0000 7fde2dc68640 1 -- 172.21.15.27:0/3871505633 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fde281340b0 con 0x7fde2812c4c0 2024-09-21T03:28:51.688 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.680+0000 7f08beffd640 1 -- 172.21.15.27:0/838016413 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c003f020 con 0x7f08d0128e10 2024-09-21T03:28:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.680+0000 7f08beffd640 1 -- 172.21.15.27:0/838016413 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f08c0003d50 con 0x7f08d0128e10 2024-09-21T03:28:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.680+0000 7f08beffd640 1 -- 172.21.15.27:0/838016413 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0037040 con 0x7f08d0128e10 2024-09-21T03:28:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.680+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08d0091350 con 0x7f08d0128e10 2024-09-21T03:28:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.680+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08d01c01c0 con 0x7f08d0128e10 2024-09-21T03:28:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.681+0000 7fde0ffff640 1 -- 172.21.15.27:0/3871505633 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7fde1c010830 con 0x7fde2812c4c0 2024-09-21T03:28:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.681+0000 7fde2dc68640 1 -- 172.21.15.27:0/3871505633 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde28134310 con 0x7fde2812c4c0 2024-09-21T03:28:51.689 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.682+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08d0071860 con 0x7f08d0128e10 2024-09-21T03:28:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.682+0000 7fde0ffff640 1 -- 172.21.15.27:0/3871505633 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7fde1c0699e0 con 0x7fde2812c4c0 2024-09-21T03:28:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.682+0000 7f08beffd640 1 -- 172.21.15.27:0/838016413 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f08c00326a0 con 0x7f08d0128e10 2024-09-21T03:28:51.690 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.683+0000 7f08beffd640 1 -- 172.21.15.27:0/838016413 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f08c00887e0 con 0x7f08d0128e10 2024-09-21T03:28:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.690+0000 7fde0ffff640 1 -- 172.21.15.27:0/3871505633 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fde1c068a90 con 0x7fde2812c4c0 2024-09-21T03:28:51.691 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.690+0000 7f08beffd640 1 -- 172.21.15.27:0/838016413 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f08c0049060 con 0x7f08d0128e10 2024-09-21T03:28:51.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.815+0000 7f09f591b640 1 -- 172.21.15.27:0/117368533 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f09f01c9ba0 con 0x7f09f012d570 2024-09-21T03:28:51.817 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.815+0000 7f09d7fff640 1 -- 172.21.15.27:0/117368533 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f09e4068490 con 0x7f09f012d570 2024-09-21T03:28:51.817 INFO:teuthology.orchestra.run.smithi027.stdout:124554051588 2024-09-21T03:28:51.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.818+0000 7f09f591b640 1 -- 172.21.15.27:0/117368533 >> v1:172.21.15.27:6800/2405211194 conn(0x7f09dc0514b0 legacy=0x7f09dc0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.820 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.818+0000 7f09f591b640 1 -- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f012d570 msgr2=0x7f09f0134410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.818+0000 7f09f591b640 1 --2- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f012d570 0x7f09f0134410 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f09e400e790 tx=0x7f09e400ec50 comp rx=0 tx=0).stop 2024-09-21T03:28:51.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.818+0000 7f09f591b640 1 -- 172.21.15.27:0/117368533 shutdown_connections 2024-09-21T03:28:51.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.818+0000 7f09f591b640 1 --2- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f09f012dec0 0x7f09f0134950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.818+0000 7f09f591b640 1 --2- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f012d570 0x7f09f0134410 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.818+0000 7f09f591b640 1 --2- 172.21.15.27:0/117368533 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f09f0094580 0x7f09f0133ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.818+0000 7f09f591b640 1 -- 172.21.15.27:0/117368533 >> 172.21.15.27:0/117368533 conn(0x7f09f009e5e0 msgr2=0x7f09f012b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:51.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.819+0000 7f09f591b640 1 -- 172.21.15.27:0/117368533 shutdown_connections 2024-09-21T03:28:51.821 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.819+0000 7f09f591b640 1 -- 172.21.15.27:0/117368533 wait complete. 2024-09-21T03:28:51.836 INFO:tasks.ceph.ceph_manager.ceph:need seq 124554051588 got 124554051588 for osd.2 2024-09-21T03:28:51.836 DEBUG:teuthology.parallel:result is None 2024-09-21T03:28:51.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.852+0000 7fde2dc68640 1 -- 172.21.15.27:0/3871505633 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fde28071860 con 0x7fde2812c4c0 2024-09-21T03:28:51.854 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.853+0000 7fde0ffff640 1 -- 172.21.15.27:0/3871505633 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fde1c0685d0 con 0x7fde2812c4c0 2024-09-21T03:28:51.854 INFO:teuthology.orchestra.run.smithi027.stdout:133143986179 2024-09-21T03:28:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.853+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f08d006dda0 con 0x7f08d0128e10 2024-09-21T03:28:51.855 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.854+0000 7f08beffd640 1 -- 172.21.15.27:0/838016413 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f08c0066020 con 0x7f08d0128e10 2024-09-21T03:28:51.855 INFO:teuthology.orchestra.run.smithi027.stdout:115964116996 2024-09-21T03:28:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.855+0000 7fde2dc68640 1 -- 172.21.15.27:0/3871505633 >> v1:172.21.15.27:6800/2405211194 conn(0x7fde140514e0 legacy=0x7fde140539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.856 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.855+0000 7fde2dc68640 1 -- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fde2812c4c0 msgr2=0x7fde28097450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.855+0000 7fde2dc68640 1 --2- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fde2812c4c0 0x7fde28097450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde1c00e6d0 tx=0x7fde1c00eb90 comp rx=0 tx=0).stop 2024-09-21T03:28:51.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.855+0000 7fde2dc68640 1 -- 172.21.15.27:0/3871505633 shutdown_connections 2024-09-21T03:28:51.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.855+0000 7fde2dc68640 1 --2- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fde2812c9f0 0x7fde28097990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.855+0000 7fde2dc68640 1 --2- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fde2812c4c0 0x7fde28097450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.855+0000 7fde2dc68640 1 --2- 172.21.15.27:0/3871505633 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fde28128e10 0x7fde28096f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.855+0000 7fde2dc68640 1 -- 172.21.15.27:0/3871505633 >> 172.21.15.27:0/3871505633 conn(0x7fde2809eb80 msgr2=0x7fde28123a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:51.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.855+0000 7fde2dc68640 1 -- 172.21.15.27:0/3871505633 shutdown_connections 2024-09-21T03:28:51.857 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.855+0000 7fde2dc68640 1 -- 172.21.15.27:0/3871505633 wait complete. 2024-09-21T03:28:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.856+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 >> v1:172.21.15.27:6800/2405211194 conn(0x7f08b0051500 legacy=0x7f08b0053a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.856+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d0128e10 msgr2=0x7f08d01c12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.856+0000 7f08d4813640 1 --2- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d0128e10 0x7f08d01c12b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f08d0072300 tx=0x7f08c0003b60 comp rx=0 tx=0).stop 2024-09-21T03:28:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.856+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 shutdown_connections 2024-09-21T03:28:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.856+0000 7f08d4813640 1 --2- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f08d012c9f0 0x7f08d01c1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.856+0000 7f08d4813640 1 --2- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f08d012c4c0 0x7f08d01c17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.856+0000 7f08d4813640 1 --2- 172.21.15.27:0/838016413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f08d0128e10 0x7f08d01c12b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.856+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 >> 172.21.15.27:0/838016413 conn(0x7f08d009eb80 msgr2=0x7f08d009b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:51.858 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.856+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 shutdown_connections 2024-09-21T03:28:51.859 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.856+0000 7f08d4813640 1 -- 172.21.15.27:0/838016413 wait complete. 2024-09-21T03:28:51.869 INFO:tasks.ceph.ceph_manager.ceph:need seq 115964116996 got 115964116996 for osd.0 2024-09-21T03:28:51.870 DEBUG:teuthology.parallel:result is None 2024-09-21T03:28:51.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.870+0000 7ff2dee6f640 1 -- 172.21.15.27:0/345740323 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ff2d806dda0 con 0x7ff2d81296f0 2024-09-21T03:28:51.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.870+0000 7ff2d57fa640 1 -- 172.21.15.27:0/345740323 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff2c40093f0 con 0x7ff2d81296f0 2024-09-21T03:28:51.872 INFO:teuthology.orchestra.run.smithi027.stdout:120259084292 2024-09-21T03:28:51.873 INFO:tasks.ceph.ceph_manager.ceph:need seq 133143986179 got 133143986179 for osd.3 2024-09-21T03:28:51.873 DEBUG:teuthology.parallel:result is None 2024-09-21T03:28:51.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.872+0000 7ff2dee6f640 1 -- 172.21.15.27:0/345740323 >> v1:172.21.15.27:6800/2405211194 conn(0x7ff2bc0514b0 legacy=0x7ff2bc0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.872+0000 7ff2dee6f640 1 -- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff2d81296f0 msgr2=0x7ff2d80985c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:51.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.872+0000 7ff2dee6f640 1 --2- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff2d81296f0 0x7ff2d80985c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400e9b0 tx=0x7ff2c400ee70 comp rx=0 tx=0).stop 2024-09-21T03:28:51.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.873+0000 7ff2dee6f640 1 -- 172.21.15.27:0/345740323 shutdown_connections 2024-09-21T03:28:51.874 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.873+0000 7ff2dee6f640 1 --2- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff2d812c8f0 0x7ff2d8098b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.873+0000 7ff2dee6f640 1 --2- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ff2d81296f0 0x7ff2d80985c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.873+0000 7ff2dee6f640 1 --2- 172.21.15.27:0/345740323 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ff2d8072980 0x7ff2d8098080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.873+0000 7ff2dee6f640 1 -- 172.21.15.27:0/345740323 >> 172.21.15.27:0/345740323 conn(0x7ff2d809edc0 msgr2=0x7ff2d81346d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.873+0000 7ff2dee6f640 1 -- 172.21.15.27:0/345740323 shutdown_connections 2024-09-21T03:28:51.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:51.873+0000 7ff2dee6f640 1 -- 172.21.15.27:0/345740323 wait complete. 2024-09-21T03:28:51.885 INFO:tasks.ceph.ceph_manager.ceph:need seq 120259084292 got 120259084292 for osd.1 2024-09-21T03:28:51.885 DEBUG:teuthology.parallel:result is None 2024-09-21T03:28:51.885 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T03:28:51.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T03:28:52.040 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.039+0000 7efc08220640 1 -- 172.21.15.27:0/903550102 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efc00135100 msgr2=0x7efc00137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.039+0000 7efc08220640 1 --2- 172.21.15.27:0/903550102 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efc00135100 0x7efc00137520 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efbf8009be0 tx=0x7efbf802a770 comp rx=0 tx=0).stop 2024-09-21T03:28:52.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.039+0000 7efc08220640 1 -- 172.21.15.27:0/903550102 shutdown_connections 2024-09-21T03:28:52.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.039+0000 7efc08220640 1 --2- 172.21.15.27:0/903550102 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efc00135100 0x7efc00137520 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.039+0000 7efc08220640 1 --2- 172.21.15.27:0/903550102 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efc000705f0 0x7efc00134bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.039+0000 7efc08220640 1 -- 172.21.15.27:0/903550102 >> 172.21.15.27:0/903550102 conn(0x7efc0009fe40 msgr2=0x7efc000a0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:52.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.039+0000 7efc08220640 1 -- 172.21.15.27:0/903550102 shutdown_connections 2024-09-21T03:28:52.041 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.039+0000 7efc08220640 1 -- 172.21.15.27:0/903550102 wait complete. 2024-09-21T03:28:52.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.040+0000 7efc08220640 1 Processor -- start 2024-09-21T03:28:52.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.040+0000 7efc08220640 1 -- start start 2024-09-21T03:28:52.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc08220640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0006fad0 0x7efc00097620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:52.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc08220640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efc000705f0 0x7efc00097b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:52.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc08220640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efc00135100 0x7efc000980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:52.043 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc08220640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7efc00138590 con 0x7efc0006fad0 2024-09-21T03:28:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc08220640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7efc00138410 con 0x7efc000705f0 2024-09-21T03:28:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc08220640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7efc00138710 con 0x7efc00135100 2024-09-21T03:28:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc057bb640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efc000705f0 0x7efc00097b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc05fbc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0006fad0 0x7efc00097620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc057bb640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efc000705f0 0x7efc00097b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:40428/0 (socket says 172.21.15.27:40428) 2024-09-21T03:28:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc057bb640 1 -- 172.21.15.27:0/2401307522 learned_addr learned my addr 172.21.15.27:0/2401307522 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc067bd640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efc00135100 0x7efc000980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc057bb640 1 -- 172.21.15.27:0/2401307522 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efc00135100 msgr2=0x7efc000980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc057bb640 1 --2- 172.21.15.27:0/2401307522 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efc00135100 0x7efc000980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.044 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc057bb640 1 -- 172.21.15.27:0/2401307522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0006fad0 msgr2=0x7efc00097620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc057bb640 1 --2- 172.21.15.27:0/2401307522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0006fad0 0x7efc00097620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc057bb640 1 -- 172.21.15.27:0/2401307522 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc00134e10 con 0x7efc000705f0 2024-09-21T03:28:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc05fbc640 1 --2- 172.21.15.27:0/2401307522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0006fad0 0x7efc00097620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efc057bb640 1 --2- 172.21.15.27:0/2401307522 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efc000705f0 0x7efc00097b60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efbf000e9b0 tx=0x7efbf000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.041+0000 7efbf67fc640 1 -- 172.21.15.27:0/2401307522 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf000cd80 con 0x7efc000705f0 2024-09-21T03:28:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.042+0000 7efbf67fc640 1 -- 172.21.15.27:0/2401307522 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7efbf0010430 con 0x7efc000705f0 2024-09-21T03:28:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.042+0000 7efc08220640 1 -- 172.21.15.27:0/2401307522 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc00125b90 con 0x7efc000705f0 2024-09-21T03:28:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.042+0000 7efbf67fc640 1 -- 172.21.15.27:0/2401307522 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf0010860 con 0x7efc000705f0 2024-09-21T03:28:52.046 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.042+0000 7efc08220640 1 -- 172.21.15.27:0/2401307522 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc001cba40 con 0x7efc000705f0 2024-09-21T03:28:52.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.042+0000 7efc08220640 1 -- 172.21.15.27:0/2401307522 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc0006fee0 con 0x7efc000705f0 2024-09-21T03:28:52.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.043+0000 7efbf67fc640 1 -- 172.21.15.27:0/2401307522 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7efbf001f020 con 0x7efc000705f0 2024-09-21T03:28:52.047 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.043+0000 7efbf67fc640 1 -- 172.21.15.27:0/2401307522 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7efbf00697e0 con 0x7efc000705f0 2024-09-21T03:28:52.048 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.047+0000 7efbf67fc640 1 -- 172.21.15.27:0/2401307522 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7efbf0068890 con 0x7efc000705f0 2024-09-21T03:28:52.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.200+0000 7efc08220640 1 -- 172.21.15.27:0/2401307522 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efc0006a2e0 con 0x7efbe40514e0 2024-09-21T03:28:52.202 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.201+0000 7efbf67fc640 1 -- 172.21.15.27:0/2401307522 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+32004 (unknown 2965378022 0 2565918576) 0x7efc0006a2e0 con 0x7efbe40514e0 2024-09-21T03:28:52.203 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:52.203 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-21T03:28:52.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.203+0000 7efc08220640 1 -- 172.21.15.27:0/2401307522 >> v1:172.21.15.27:6800/2405211194 conn(0x7efbe40514e0 legacy=0x7efbe40539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.203+0000 7efc08220640 1 -- 172.21.15.27:0/2401307522 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efc000705f0 msgr2=0x7efc00097b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.203+0000 7efc08220640 1 --2- 172.21.15.27:0/2401307522 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efc000705f0 0x7efc00097b60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7efbf000e9b0 tx=0x7efbf000ee70 comp rx=0 tx=0).stop 2024-09-21T03:28:52.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.203+0000 7efc08220640 1 -- 172.21.15.27:0/2401307522 shutdown_connections 2024-09-21T03:28:52.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.203+0000 7efc08220640 1 --2- 172.21.15.27:0/2401307522 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7efc00135100 0x7efc000980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.203+0000 7efc08220640 1 --2- 172.21.15.27:0/2401307522 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7efc000705f0 0x7efc00097b60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.203+0000 7efc08220640 1 --2- 172.21.15.27:0/2401307522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efc0006fad0 0x7efc00097620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.203+0000 7efc08220640 1 -- 172.21.15.27:0/2401307522 >> 172.21.15.27:0/2401307522 conn(0x7efc0009fe40 msgr2=0x7efc00135840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:52.205 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.204+0000 7efc08220640 1 -- 172.21.15.27:0/2401307522 shutdown_connections 2024-09-21T03:28:52.206 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.204+0000 7efc08220640 1 -- 172.21.15.27:0/2401307522 wait complete. 2024-09-21T03:28:52.216 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-09-21T03:28:50.299790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":151,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":116608,"kb_used_data":2280,"kb_used_omap":17,"kb_used_meta":114222,"kb_avail":377370752,"statfs":{"total":386547056640,"available":386427650048,"internally_reserved":0,"allocated":2334720,"data_stored":1576729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17729,"internal_metadata":116964031},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.454996"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982313+0000","last_change":"2024-09-21T03:28:44.676220+0000","last_active":"2024-09-21T03:28:47.982313+0000","last_peered":"2024-09-21T03:28:47.982313+0000","last_clean":"2024-09-21T03:28:47.982313+0000","last_became_active":"2024-09-21T03:28:44.675664+0000","last_became_peered":"2024-09-21T03:28:44.675664+0000","last_unstale":"2024-09-21T03:28:47.982313+0000","last_undegraded":"2024-09-21T03:28:47.982313+0000","last_fullsized":"2024-09-21T03:28:47.982313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:02.081362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393285,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249352+0000","last_change":"2024-09-21T03:28:40.835755+0000","last_active":"2024-09-21T03:28:49.249352+0000","last_peered":"2024-09-21T03:28:49.249352+0000","last_clean":"2024-09-21T03:28:49.249352+0000","last_became_active":"2024-09-21T03:28:40.835171+0000","last_became_peered":"2024-09-21T03:28:40.835171+0000","last_unstale":"2024-09-21T03:28:49.249352+0000","last_undegraded":"2024-09-21T03:28:49.249352+0000","last_fullsized":"2024-09-21T03:28:49.249352+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:17:17.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044937200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982197+0000","last_change":"2024-09-21T03:28:44.676192+0000","last_active":"2024-09-21T03:28:47.982197+0000","last_peered":"2024-09-21T03:28:47.982197+0000","last_clean":"2024-09-21T03:28:47.982197+0000","last_became_active":"2024-09-21T03:28:44.675680+0000","last_became_peered":"2024-09-21T03:28:44.675680+0000","last_unstale":"2024-09-21T03:28:47.982197+0000","last_undegraded":"2024-09-21T03:28:47.982197+0000","last_fullsized":"2024-09-21T03:28:47.982197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:21:10.791750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249310+0000","last_change":"2024-09-21T03:28:40.835586+0000","last_active":"2024-09-21T03:28:49.249310+0000","last_peered":"2024-09-21T03:28:49.249310+0000","last_clean":"2024-09-21T03:28:49.249310+0000","last_became_active":"2024-09-21T03:28:40.835243+0000","last_became_peered":"2024-09-21T03:28:40.835243+0000","last_unstale":"2024-09-21T03:28:49.249310+0000","last_undegraded":"2024-09-21T03:28:49.249310+0000","last_fullsized":"2024-09-21T03:28:49.249310+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:31.997034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049584300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":47,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982077+0000","last_change":"2024-09-21T03:28:43.852388+0000","last_active":"2024-09-21T03:28:47.982077+0000","last_peered":"2024-09-21T03:28:47.982077+0000","last_clean":"2024-09-21T03:28:47.982077+0000","last_became_active":"2024-09-21T03:28:43.851555+0000","last_became_peered":"2024-09-21T03:28:43.851555+0000","last_unstale":"2024-09-21T03:28:47.982077+0000","last_undegraded":"2024-09-21T03:28:47.982077+0000","last_fullsized":"2024-09-21T03:28:47.982077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:37:19.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010094349999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:48.996126+0000","last_change":"2024-09-21T03:28:43.850674+0000","last_active":"2024-09-21T03:28:48.996126+0000","last_peered":"2024-09-21T03:28:48.996126+0000","last_clean":"2024-09-21T03:28:48.996126+0000","last_became_active":"2024-09-21T03:28:43.850058+0000","last_became_peered":"2024-09-21T03:28:43.850058+0000","last_unstale":"2024-09-21T03:28:48.996126+0000","last_undegraded":"2024-09-21T03:28:48.996126+0000","last_fullsized":"2024-09-21T03:28:48.996126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:58:47.763461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056217599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.853213+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.851256+0000","last_became_peered":"2024-09-21T03:28:43.851256+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:03.534378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045280299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":46,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.850214+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.849169+0000","last_became_peered":"2024-09-21T03:28:43.849169+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:34.860226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'104","reported_seq":201,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249233+0000","last_change":"2024-09-21T03:28:40.835338+0000","last_active":"2024-09-21T03:28:49.249233+0000","last_peered":"2024-09-21T03:28:49.249233+0000","last_clean":"2024-09-21T03:28:49.249233+0000","last_became_active":"2024-09-21T03:28:40.834765+0000","last_became_peered":"2024-09-21T03:28:40.834765+0000","last_unstale":"2024-09-21T03:28:49.249233+0000","last_undegraded":"2024-09-21T03:28:49.249233+0000","last_fullsized":"2024-09-21T03:28:49.249233+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_clean_scrub_stamp":"2024-09-21T03:27:08.233475+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:57.980901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":31,"seq":133143986179,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":276,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":282624,"data_stored":96334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":28176047},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":29,"seq":124554051588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":280,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":286720,"data_stored":99880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":864,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":884736,"data_stored":690267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32435888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":27,"seq":115964116996,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":860,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":880640,"data_stored":690248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T03:28:52.217 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T03:28:52.331 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.329+0000 7fd35d551640 1 -- 172.21.15.27:0/3782701778 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd358070ec0 msgr2=0x7fd3581348b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.329+0000 7fd35d551640 1 --2- 172.21.15.27:0/3782701778 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd358070ec0 0x7fd3581348b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd34c009860 tx=0x7fd34c02a460 comp rx=0 tx=0).stop 2024-09-21T03:28:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.329+0000 7fd35d551640 1 -- 172.21.15.27:0/3782701778 shutdown_connections 2024-09-21T03:28:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.329+0000 7fd35d551640 1 --2- 172.21.15.27:0/3782701778 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd358134df0 0x7fd358137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.329+0000 7fd35d551640 1 --2- 172.21.15.27:0/3782701778 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd358070ec0 0x7fd3581348b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.329+0000 7fd35d551640 1 -- 172.21.15.27:0/3782701778 >> 172.21.15.27:0/3782701778 conn(0x7fd3580a2050 msgr2=0x7fd3580a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.329+0000 7fd35d551640 1 -- 172.21.15.27:0/3782701778 shutdown_connections 2024-09-21T03:28:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.329+0000 7fd35d551640 1 -- 172.21.15.27:0/3782701778 wait complete. 2024-09-21T03:28:52.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.330+0000 7fd35d551640 1 Processor -- start 2024-09-21T03:28:52.333 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.330+0000 7fd35d551640 1 -- start start 2024-09-21T03:28:52.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd35d551640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd358070ec0 0x7fd35809d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:52.334 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd35d551640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd35812f630 0x7fd35809db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd35d551640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd358134df0 0x7fd35809e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd35d551640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fd358138590 con 0x7fd358070ec0 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd35d551640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fd358138410 con 0x7fd35812f630 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd35d551640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fd358138710 con 0x7fd358134df0 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd356ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd358070ec0 0x7fd35809d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd3567fc640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd35812f630 0x7fd35809db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd3577fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd358134df0 0x7fd35809e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd356ffd640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd358070ec0 0x7fd35809d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44180/0 (socket says 172.21.15.27:44180) 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd356ffd640 1 -- 172.21.15.27:0/3529883079 learned_addr learned my addr 172.21.15.27:0/3529883079 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd3577fe640 1 -- 172.21.15.27:0/3529883079 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd35812f630 msgr2=0x7fd35809db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd3577fe640 1 --2- 172.21.15.27:0/3529883079 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd35812f630 0x7fd35809db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd3577fe640 1 -- 172.21.15.27:0/3529883079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd358070ec0 msgr2=0x7fd35809d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd3577fe640 1 --2- 172.21.15.27:0/3529883079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd358070ec0 0x7fd35809d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd3577fe640 1 -- 172.21.15.27:0/3529883079 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3581306b0 con 0x7fd358134df0 2024-09-21T03:28:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.331+0000 7fd356ffd640 1 --2- 172.21.15.27:0/3529883079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd358070ec0 0x7fd35809d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.332+0000 7fd3577fe640 1 --2- 172.21.15.27:0/3529883079 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd358134df0 0x7fd35809e040 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd35000eb50 tx=0x7fd35000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.332+0000 7fd33f7fe640 1 -- 172.21.15.27:0/3529883079 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350003b70 con 0x7fd358134df0 2024-09-21T03:28:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.332+0000 7fd33f7fe640 1 -- 172.21.15.27:0/3529883079 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fd350003d10 con 0x7fd358134df0 2024-09-21T03:28:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.332+0000 7fd35d551640 1 -- 172.21.15.27:0/3529883079 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3581317e0 con 0x7fd358134df0 2024-09-21T03:28:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.332+0000 7fd33f7fe640 1 -- 172.21.15.27:0/3529883079 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350004280 con 0x7fd358134df0 2024-09-21T03:28:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.332+0000 7fd35d551640 1 -- 172.21.15.27:0/3529883079 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fd358131440 con 0x7fd358134df0 2024-09-21T03:28:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.333+0000 7fd35d551640 1 -- 172.21.15.27:0/3529883079 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd35812fb60 con 0x7fd358134df0 2024-09-21T03:28:52.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.333+0000 7fd33f7fe640 1 -- 172.21.15.27:0/3529883079 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7fd350004420 con 0x7fd358134df0 2024-09-21T03:28:52.337 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.333+0000 7fd33f7fe640 1 -- 172.21.15.27:0/3529883079 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7fd350069b60 con 0x7fd358134df0 2024-09-21T03:28:52.339 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.337+0000 7fd33f7fe640 1 -- 172.21.15.27:0/3529883079 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fd35000ee50 con 0x7fd358134df0 2024-09-21T03:28:52.496 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.494+0000 7fd35d551640 1 -- 172.21.15.27:0/3529883079 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd35806dda0 con 0x7fd3440514b0 2024-09-21T03:28:52.497 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.495+0000 7fd33f7fe640 1 -- 172.21.15.27:0/3529883079 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+32004 (unknown 2965378022 0 729647667) 0x7fd35806dda0 con 0x7fd3440514b0 2024-09-21T03:28:52.497 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:52.498 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-21T03:28:52.499 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.496+0000 7fd35d551640 1 -- 172.21.15.27:0/3529883079 >> v1:172.21.15.27:6800/2405211194 conn(0x7fd3440514b0 legacy=0x7fd3440539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.496+0000 7fd35d551640 1 -- 172.21.15.27:0/3529883079 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd358134df0 msgr2=0x7fd35809e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.497+0000 7fd35d551640 1 --2- 172.21.15.27:0/3529883079 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd358134df0 0x7fd35809e040 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd35000eb50 tx=0x7fd35000c6a0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.497+0000 7fd35d551640 1 -- 172.21.15.27:0/3529883079 shutdown_connections 2024-09-21T03:28:52.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.497+0000 7fd35d551640 1 --2- 172.21.15.27:0/3529883079 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fd358134df0 0x7fd35809e040 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.497+0000 7fd35d551640 1 --2- 172.21.15.27:0/3529883079 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fd35812f630 0x7fd35809db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.497+0000 7fd35d551640 1 --2- 172.21.15.27:0/3529883079 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd358070ec0 0x7fd35809d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.497+0000 7fd35d551640 1 -- 172.21.15.27:0/3529883079 >> 172.21.15.27:0/3529883079 conn(0x7fd3580a2050 msgr2=0x7fd358073860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:52.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.497+0000 7fd35d551640 1 -- 172.21.15.27:0/3529883079 shutdown_connections 2024-09-21T03:28:52.500 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.497+0000 7fd35d551640 1 -- 172.21.15.27:0/3529883079 wait complete. 2024-09-21T03:28:52.510 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-21T03:28:52.300195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":151,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":116608,"kb_used_data":2280,"kb_used_omap":17,"kb_used_meta":114222,"kb_avail":377370752,"statfs":{"total":386547056640,"available":386427650048,"internally_reserved":0,"allocated":2334720,"data_stored":1576729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17729,"internal_metadata":116964031},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001795"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982313+0000","last_change":"2024-09-21T03:28:44.676220+0000","last_active":"2024-09-21T03:28:47.982313+0000","last_peered":"2024-09-21T03:28:47.982313+0000","last_clean":"2024-09-21T03:28:47.982313+0000","last_became_active":"2024-09-21T03:28:44.675664+0000","last_became_peered":"2024-09-21T03:28:44.675664+0000","last_unstale":"2024-09-21T03:28:47.982313+0000","last_undegraded":"2024-09-21T03:28:47.982313+0000","last_fullsized":"2024-09-21T03:28:47.982313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:02.081362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393285,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249352+0000","last_change":"2024-09-21T03:28:40.835755+0000","last_active":"2024-09-21T03:28:49.249352+0000","last_peered":"2024-09-21T03:28:49.249352+0000","last_clean":"2024-09-21T03:28:49.249352+0000","last_became_active":"2024-09-21T03:28:40.835171+0000","last_became_peered":"2024-09-21T03:28:40.835171+0000","last_unstale":"2024-09-21T03:28:49.249352+0000","last_undegraded":"2024-09-21T03:28:49.249352+0000","last_fullsized":"2024-09-21T03:28:49.249352+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:17:17.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044937200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982197+0000","last_change":"2024-09-21T03:28:44.676192+0000","last_active":"2024-09-21T03:28:47.982197+0000","last_peered":"2024-09-21T03:28:47.982197+0000","last_clean":"2024-09-21T03:28:47.982197+0000","last_became_active":"2024-09-21T03:28:44.675680+0000","last_became_peered":"2024-09-21T03:28:44.675680+0000","last_unstale":"2024-09-21T03:28:47.982197+0000","last_undegraded":"2024-09-21T03:28:47.982197+0000","last_fullsized":"2024-09-21T03:28:47.982197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:21:10.791750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249310+0000","last_change":"2024-09-21T03:28:40.835586+0000","last_active":"2024-09-21T03:28:49.249310+0000","last_peered":"2024-09-21T03:28:49.249310+0000","last_clean":"2024-09-21T03:28:49.249310+0000","last_became_active":"2024-09-21T03:28:40.835243+0000","last_became_peered":"2024-09-21T03:28:40.835243+0000","last_unstale":"2024-09-21T03:28:49.249310+0000","last_undegraded":"2024-09-21T03:28:49.249310+0000","last_fullsized":"2024-09-21T03:28:49.249310+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:31.997034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049584300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":47,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982077+0000","last_change":"2024-09-21T03:28:43.852388+0000","last_active":"2024-09-21T03:28:47.982077+0000","last_peered":"2024-09-21T03:28:47.982077+0000","last_clean":"2024-09-21T03:28:47.982077+0000","last_became_active":"2024-09-21T03:28:43.851555+0000","last_became_peered":"2024-09-21T03:28:43.851555+0000","last_unstale":"2024-09-21T03:28:47.982077+0000","last_undegraded":"2024-09-21T03:28:47.982077+0000","last_fullsized":"2024-09-21T03:28:47.982077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:37:19.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010094349999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:48.996126+0000","last_change":"2024-09-21T03:28:43.850674+0000","last_active":"2024-09-21T03:28:48.996126+0000","last_peered":"2024-09-21T03:28:48.996126+0000","last_clean":"2024-09-21T03:28:48.996126+0000","last_became_active":"2024-09-21T03:28:43.850058+0000","last_became_peered":"2024-09-21T03:28:43.850058+0000","last_unstale":"2024-09-21T03:28:48.996126+0000","last_undegraded":"2024-09-21T03:28:48.996126+0000","last_fullsized":"2024-09-21T03:28:48.996126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:58:47.763461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056217599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.853213+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.851256+0000","last_became_peered":"2024-09-21T03:28:43.851256+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:03.534378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045280299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":46,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.850214+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.849169+0000","last_became_peered":"2024-09-21T03:28:43.849169+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:34.860226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'104","reported_seq":201,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249233+0000","last_change":"2024-09-21T03:28:40.835338+0000","last_active":"2024-09-21T03:28:49.249233+0000","last_peered":"2024-09-21T03:28:49.249233+0000","last_clean":"2024-09-21T03:28:49.249233+0000","last_became_active":"2024-09-21T03:28:40.834765+0000","last_became_peered":"2024-09-21T03:28:40.834765+0000","last_unstale":"2024-09-21T03:28:49.249233+0000","last_undegraded":"2024-09-21T03:28:49.249233+0000","last_fullsized":"2024-09-21T03:28:49.249233+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_clean_scrub_stamp":"2024-09-21T03:27:08.233475+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:57.980901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":31,"seq":133143986180,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":276,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":282624,"data_stored":96334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":28176047},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":29,"seq":124554051588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":280,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":286720,"data_stored":99880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":864,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":884736,"data_stored":690267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32435888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":27,"seq":115964116996,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":860,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":880640,"data_stored":690248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T03:28:52.511 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T03:28:52.511 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T03:28:52.511 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T03:28:52.511 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T03:28:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.622+0000 7f3710339640 1 -- 172.21.15.27:0/301696148 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f37081353f0 msgr2=0x7f3708137810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.622+0000 7f3710339640 1 --2- 172.21.15.27:0/301696148 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f37081353f0 0x7f3708137810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3700005fd0 tx=0x7f370002a820 comp rx=0 tx=0).stop 2024-09-21T03:28:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.623+0000 7f3710339640 1 -- 172.21.15.27:0/301696148 shutdown_connections 2024-09-21T03:28:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.623+0000 7f3710339640 1 --2- 172.21.15.27:0/301696148 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f37081353f0 0x7f3708137810 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.625 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.623+0000 7f3710339640 1 --2- 172.21.15.27:0/301696148 >> v2:172.21.15.27:3300/0 conn(0x7f3708130790 0x7f3708134eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.623+0000 7f3710339640 1 -- 172.21.15.27:0/301696148 >> 172.21.15.27:0/301696148 conn(0x7f37080a1f40 msgr2=0x7f37080a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:52.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.623+0000 7f3710339640 1 -- 172.21.15.27:0/301696148 shutdown_connections 2024-09-21T03:28:52.626 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.623+0000 7f3710339640 1 -- 172.21.15.27:0/301696148 wait complete. 2024-09-21T03:28:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f3710339640 1 Processor -- start 2024-09-21T03:28:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f3710339640 1 -- start start 2024-09-21T03:28:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f3710339640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f370812fc70 0x7f37081c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f3710339640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3708130790 0x7f37081c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f3710339640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37081353f0 0x7f37081c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f3710339640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3708138870 con 0x7f37081353f0 2024-09-21T03:28:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f3710339640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f37081386f0 con 0x7f370812fc70 2024-09-21T03:28:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f3710339640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f37081389f0 con 0x7f3708130790 2024-09-21T03:28:52.628 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f370e0d5640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f370812fc70 0x7f37081c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f370e8d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37081353f0 0x7f37081c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f370d8d4640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3708130790 0x7f37081c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f370d8d4640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3708130790 0x7f37081c6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:35572/0 (socket says 172.21.15.27:35572) 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.624+0000 7f370e8d6640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37081353f0 0x7f37081c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44204/0 (socket says 172.21.15.27:44204) 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f370d8d4640 1 -- 172.21.15.27:0/2918290934 learned_addr learned my addr 172.21.15.27:0/2918290934 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f370e8d6640 1 -- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3708130790 msgr2=0x7f37081c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f370e8d6640 1 --2- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3708130790 0x7f37081c6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f370e8d6640 1 -- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f370812fc70 msgr2=0x7f37081c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f370e8d6640 1 --2- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f370812fc70 0x7f37081c5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f370e8d6640 1 -- 172.21.15.27:0/2918290934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37080915b0 con 0x7f37081353f0 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f370d8d4640 1 --2- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3708130790 0x7f37081c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f370e0d5640 1 --2- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f370812fc70 0x7f37081c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f370e8d6640 1 --2- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37081353f0 0x7f37081c6740 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3700009000 tx=0x7f3700032d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:52.629 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f36feffd640 1 -- 172.21.15.27:0/2918290934 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700037040 con 0x7f37081353f0 2024-09-21T03:28:52.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f36feffd640 1 -- 172.21.15.27:0/2918290934 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3700042070 con 0x7f37081353f0 2024-09-21T03:28:52.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f36feffd640 1 -- 172.21.15.27:0/2918290934 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37000042a0 con 0x7f37081353f0 2024-09-21T03:28:52.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f3710339640 1 -- 172.21.15.27:0/2918290934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3708091000 con 0x7f37081353f0 2024-09-21T03:28:52.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.625+0000 7f3710339640 1 -- 172.21.15.27:0/2918290934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3708133390 con 0x7f37081353f0 2024-09-21T03:28:52.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.626+0000 7f3710339640 1 -- 172.21.15.27:0/2918290934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37080a08c0 con 0x7f37081353f0 2024-09-21T03:28:52.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.627+0000 7f36feffd640 1 -- 172.21.15.27:0/2918290934 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f3700004440 con 0x7f37081353f0 2024-09-21T03:28:52.630 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.627+0000 7f36feffd640 1 -- 172.21.15.27:0/2918290934 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f3700089cf0 con 0x7f37081353f0 2024-09-21T03:28:52.633 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.631+0000 7f36feffd640 1 -- 172.21.15.27:0/2918290934 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f37080a08c0 con 0x7f37081353f0 2024-09-21T03:28:52.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.814+0000 7f3710339640 1 -- 172.21.15.27:0/2918290934 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f3708134a40 con 0x7f37081353f0 2024-09-21T03:28:52.816 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.814+0000 7f36feffd640 1 -- 172.21.15.27:0/2918290934 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f3708134a40 con 0x7f37081353f0 2024-09-21T03:28:52.816 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:52.816 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T03:28:52.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.816+0000 7f3710339640 1 -- 172.21.15.27:0/2918290934 >> v1:172.21.15.27:6800/2405211194 conn(0x7f36ec0514e0 legacy=0x7f36ec0539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.818 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.816+0000 7f3710339640 1 -- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37081353f0 msgr2=0x7f37081c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:52.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.816+0000 7f3710339640 1 --2- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37081353f0 0x7f37081c6740 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3700009000 tx=0x7f3700032d10 comp rx=0 tx=0).stop 2024-09-21T03:28:52.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.816+0000 7f3710339640 1 -- 172.21.15.27:0/2918290934 shutdown_connections 2024-09-21T03:28:52.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.816+0000 7f3710339640 1 --2- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37081353f0 0x7f37081c6740 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.816+0000 7f3710339640 1 --2- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3708130790 0x7f37081c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.816+0000 7f3710339640 1 --2- 172.21.15.27:0/2918290934 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f370812fc70 0x7f37081c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:52.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.816+0000 7f3710339640 1 -- 172.21.15.27:0/2918290934 >> 172.21.15.27:0/2918290934 conn(0x7f37080a1f40 msgr2=0x7f3708135800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:52.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.816+0000 7f3710339640 1 -- 172.21.15.27:0/2918290934 shutdown_connections 2024-09-21T03:28:52.819 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:52.816+0000 7f3710339640 1 -- 172.21.15.27:0/2918290934 wait complete. 2024-09-21T03:28:52.830 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T03:28:52.830 INFO:teuthology.run_tasks:Running task print... 2024-09-21T03:28:52.839 INFO:teuthology.task.print:**** done ceph.restart 2024-09-21T03:28:52.839 DEBUG:teuthology.run_tasks:Unwinding manager ceph.restart 2024-09-21T03:28:52.857 DEBUG:teuthology.run_tasks:Unwinding manager ceph.restart 2024-09-21T03:28:52.868 DEBUG:teuthology.run_tasks:Unwinding manager install.upgrade 2024-09-21T03:28:52.880 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-21T03:28:52.892 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T03:28:52.892 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T03:28:53.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.008+0000 7f531a517640 1 -- 172.21.15.27:0/3033804442 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f531412cbd0 msgr2=0x7f5314134970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.010 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.008+0000 7f531a517640 1 --2- 172.21.15.27:0/3033804442 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f531412cbd0 0x7f5314134970 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5308009900 tx=0x7f530802a5c0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.008+0000 7f531a517640 1 -- 172.21.15.27:0/3033804442 shutdown_connections 2024-09-21T03:28:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.008+0000 7f531a517640 1 --2- 172.21.15.27:0/3033804442 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5314134eb0 0x7f53141372d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.008+0000 7f531a517640 1 --2- 172.21.15.27:0/3033804442 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f531412cbd0 0x7f5314134970 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.008+0000 7f531a517640 1 -- 172.21.15.27:0/3033804442 >> 172.21.15.27:0/3033804442 conn(0x7f53140a1d70 msgr2=0x7f53140a2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.008+0000 7f531a517640 1 -- 172.21.15.27:0/3033804442 shutdown_connections 2024-09-21T03:28:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.008+0000 7f531a517640 1 -- 172.21.15.27:0/3033804442 wait complete. 2024-09-21T03:28:53.011 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.009+0000 7f531a517640 1 Processor -- start 2024-09-21T03:28:53.012 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.010+0000 7f531a517640 1 -- start start 2024-09-21T03:28:53.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.010+0000 7f531a517640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f53140718e0 0x7f531409b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.010+0000 7f531a517640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531412cbd0 0x7f531409b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f531a517640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5314134eb0 0x7f531409bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.014 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f531a517640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5314138720 con 0x7f531412cbd0 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f531a517640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f53141385a0 con 0x7f53140718e0 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f531a517640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f53141388a0 con 0x7f5314134eb0 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f53137fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531412cbd0 0x7f531409b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f5313fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f53140718e0 0x7f531409b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f53137fe640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531412cbd0 0x7f531409b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44208/0 (socket says 172.21.15.27:44208) 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f5318ab4640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5314134eb0 0x7f531409bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f5313fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f53140718e0 0x7f531409b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:46232/0 (socket says 172.21.15.27:46232) 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f53137fe640 1 -- 172.21.15.27:0/2926853472 learned_addr learned my addr 172.21.15.27:0/2926853472 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f53137fe640 1 -- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5314134eb0 msgr2=0x7f531409bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f53137fe640 1 --2- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5314134eb0 0x7f531409bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f53137fe640 1 -- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f53140718e0 msgr2=0x7f531409b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f53137fe640 1 --2- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f53140718e0 0x7f531409b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f53137fe640 1 -- 172.21.15.27:0/2926853472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5314131120 con 0x7f531412cbd0 2024-09-21T03:28:53.015 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f5318ab4640 1 --2- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5314134eb0 0x7f531409bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f5313fff640 1 --2- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f53140718e0 0x7f531409b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f53137fe640 1 --2- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531412cbd0 0x7f531409b9c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5308006010 tx=0x7f53080038a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f5310ff9640 1 -- 172.21.15.27:0/2926853472 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530803f070 con 0x7f531412cbd0 2024-09-21T03:28:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f5310ff9640 1 -- 172.21.15.27:0/2926853472 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f53080047a0 con 0x7f531412cbd0 2024-09-21T03:28:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f5310ff9640 1 -- 172.21.15.27:0/2926853472 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5308035030 con 0x7f531412cbd0 2024-09-21T03:28:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.011+0000 7f531a517640 1 -- 172.21.15.27:0/2926853472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5314131f70 con 0x7f531412cbd0 2024-09-21T03:28:53.016 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.012+0000 7f531a517640 1 -- 172.21.15.27:0/2926853472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f531409df70 con 0x7f531412cbd0 2024-09-21T03:28:53.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.013+0000 7f5310ff9640 1 -- 172.21.15.27:0/2926853472 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f5308042070 con 0x7f531412cbd0 2024-09-21T03:28:53.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.013+0000 7f531a517640 1 -- 172.21.15.27:0/2926853472 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5314070230 con 0x7f531412cbd0 2024-09-21T03:28:53.017 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.013+0000 7f5310ff9640 1 -- 172.21.15.27:0/2926853472 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f5308089f80 con 0x7f531412cbd0 2024-09-21T03:28:53.019 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.017+0000 7f5310ff9640 1 -- 172.21.15.27:0/2926853472 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f5314070230 con 0x7f531412cbd0 2024-09-21T03:28:53.176 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.174+0000 7f531a517640 1 -- 172.21.15.27:0/2926853472 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f53141314e0 con 0x7f53000514b0 2024-09-21T03:28:53.177 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.175+0000 7f5310ff9640 1 -- 172.21.15.27:0/2926853472 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+32004 (unknown 2965378022 0 729647667) 0x7f53141314e0 con 0x7f53000514b0 2024-09-21T03:28:53.178 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:53.178 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-21T03:28:53.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.177+0000 7f531a517640 1 -- 172.21.15.27:0/2926853472 >> v1:172.21.15.27:6800/2405211194 conn(0x7f53000514b0 legacy=0x7f53000539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.179 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.177+0000 7f531a517640 1 -- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531412cbd0 msgr2=0x7f531409b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.177+0000 7f531a517640 1 --2- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531412cbd0 0x7f531409b9c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5308006010 tx=0x7f53080038a0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.177+0000 7f531a517640 1 -- 172.21.15.27:0/2926853472 shutdown_connections 2024-09-21T03:28:53.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.177+0000 7f531a517640 1 --2- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5314134eb0 0x7f531409bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.177+0000 7f531a517640 1 --2- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f531412cbd0 0x7f531409b9c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.177+0000 7f531a517640 1 --2- 172.21.15.27:0/2926853472 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f53140718e0 0x7f531409b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.177+0000 7f531a517640 1 -- 172.21.15.27:0/2926853472 >> 172.21.15.27:0/2926853472 conn(0x7f53140a1d70 msgr2=0x7f531412d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:53.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.177+0000 7f531a517640 1 -- 172.21.15.27:0/2926853472 shutdown_connections 2024-09-21T03:28:53.180 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.177+0000 7f531a517640 1 -- 172.21.15.27:0/2926853472 wait complete. 2024-09-21T03:28:53.190 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-21T03:28:52.300195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":151,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":116608,"kb_used_data":2280,"kb_used_omap":17,"kb_used_meta":114222,"kb_avail":377370752,"statfs":{"total":386547056640,"available":386427650048,"internally_reserved":0,"allocated":2334720,"data_stored":1576729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17729,"internal_metadata":116964031},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001795"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982313+0000","last_change":"2024-09-21T03:28:44.676220+0000","last_active":"2024-09-21T03:28:47.982313+0000","last_peered":"2024-09-21T03:28:47.982313+0000","last_clean":"2024-09-21T03:28:47.982313+0000","last_became_active":"2024-09-21T03:28:44.675664+0000","last_became_peered":"2024-09-21T03:28:44.675664+0000","last_unstale":"2024-09-21T03:28:47.982313+0000","last_undegraded":"2024-09-21T03:28:47.982313+0000","last_fullsized":"2024-09-21T03:28:47.982313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:02.081362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393285,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249352+0000","last_change":"2024-09-21T03:28:40.835755+0000","last_active":"2024-09-21T03:28:49.249352+0000","last_peered":"2024-09-21T03:28:49.249352+0000","last_clean":"2024-09-21T03:28:49.249352+0000","last_became_active":"2024-09-21T03:28:40.835171+0000","last_became_peered":"2024-09-21T03:28:40.835171+0000","last_unstale":"2024-09-21T03:28:49.249352+0000","last_undegraded":"2024-09-21T03:28:49.249352+0000","last_fullsized":"2024-09-21T03:28:49.249352+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:17:17.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044937200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982197+0000","last_change":"2024-09-21T03:28:44.676192+0000","last_active":"2024-09-21T03:28:47.982197+0000","last_peered":"2024-09-21T03:28:47.982197+0000","last_clean":"2024-09-21T03:28:47.982197+0000","last_became_active":"2024-09-21T03:28:44.675680+0000","last_became_peered":"2024-09-21T03:28:44.675680+0000","last_unstale":"2024-09-21T03:28:47.982197+0000","last_undegraded":"2024-09-21T03:28:47.982197+0000","last_fullsized":"2024-09-21T03:28:47.982197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:21:10.791750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249310+0000","last_change":"2024-09-21T03:28:40.835586+0000","last_active":"2024-09-21T03:28:49.249310+0000","last_peered":"2024-09-21T03:28:49.249310+0000","last_clean":"2024-09-21T03:28:49.249310+0000","last_became_active":"2024-09-21T03:28:40.835243+0000","last_became_peered":"2024-09-21T03:28:40.835243+0000","last_unstale":"2024-09-21T03:28:49.249310+0000","last_undegraded":"2024-09-21T03:28:49.249310+0000","last_fullsized":"2024-09-21T03:28:49.249310+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:31.997034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049584300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":47,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982077+0000","last_change":"2024-09-21T03:28:43.852388+0000","last_active":"2024-09-21T03:28:47.982077+0000","last_peered":"2024-09-21T03:28:47.982077+0000","last_clean":"2024-09-21T03:28:47.982077+0000","last_became_active":"2024-09-21T03:28:43.851555+0000","last_became_peered":"2024-09-21T03:28:43.851555+0000","last_unstale":"2024-09-21T03:28:47.982077+0000","last_undegraded":"2024-09-21T03:28:47.982077+0000","last_fullsized":"2024-09-21T03:28:47.982077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:37:19.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010094349999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:48.996126+0000","last_change":"2024-09-21T03:28:43.850674+0000","last_active":"2024-09-21T03:28:48.996126+0000","last_peered":"2024-09-21T03:28:48.996126+0000","last_clean":"2024-09-21T03:28:48.996126+0000","last_became_active":"2024-09-21T03:28:43.850058+0000","last_became_peered":"2024-09-21T03:28:43.850058+0000","last_unstale":"2024-09-21T03:28:48.996126+0000","last_undegraded":"2024-09-21T03:28:48.996126+0000","last_fullsized":"2024-09-21T03:28:48.996126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:58:47.763461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056217599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.853213+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.851256+0000","last_became_peered":"2024-09-21T03:28:43.851256+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:03.534378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045280299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":46,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.850214+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.849169+0000","last_became_peered":"2024-09-21T03:28:43.849169+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:34.860226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'104","reported_seq":201,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249233+0000","last_change":"2024-09-21T03:28:40.835338+0000","last_active":"2024-09-21T03:28:49.249233+0000","last_peered":"2024-09-21T03:28:49.249233+0000","last_clean":"2024-09-21T03:28:49.249233+0000","last_became_active":"2024-09-21T03:28:40.834765+0000","last_became_peered":"2024-09-21T03:28:40.834765+0000","last_unstale":"2024-09-21T03:28:49.249233+0000","last_undegraded":"2024-09-21T03:28:49.249233+0000","last_fullsized":"2024-09-21T03:28:49.249233+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_clean_scrub_stamp":"2024-09-21T03:27:08.233475+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:57.980901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":31,"seq":133143986180,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":276,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":282624,"data_stored":96334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":28176047},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":29,"seq":124554051588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":280,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":286720,"data_stored":99880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":864,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":884736,"data_stored":690267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32435888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":27,"seq":115964116996,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":860,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":880640,"data_stored":690248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T03:28:53.191 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T03:28:53.305 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.303+0000 7f3e2fb65640 1 -- 172.21.15.27:0/2314998934 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3e281305c0 msgr2=0x7f3e28072830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.303+0000 7f3e2fb65640 1 --2- 172.21.15.27:0/2314998934 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3e281305c0 0x7f3e28072830 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e18005fd0 tx=0x7f3e1802a540 comp rx=0 tx=0).stop 2024-09-21T03:28:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.303+0000 7f3e2fb65640 1 -- 172.21.15.27:0/2314998934 shutdown_connections 2024-09-21T03:28:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.303+0000 7f3e2fb65640 1 --2- 172.21.15.27:0/2314998934 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3e281305c0 0x7f3e28072830 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.303+0000 7f3e2fb65640 1 --2- 172.21.15.27:0/2314998934 >> v2:172.21.15.27:3300/0 conn(0x7f3e2812fc70 0x7f3e28130080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.303+0000 7f3e2fb65640 1 -- 172.21.15.27:0/2314998934 >> 172.21.15.27:0/2314998934 conn(0x7f3e280a1f40 msgr2=0x7f3e280a2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.303+0000 7f3e2fb65640 1 -- 172.21.15.27:0/2314998934 shutdown_connections 2024-09-21T03:28:53.306 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.303+0000 7f3e2fb65640 1 -- 172.21.15.27:0/2314998934 wait complete. 2024-09-21T03:28:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.304+0000 7f3e2fb65640 1 Processor -- start 2024-09-21T03:28:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.304+0000 7f3e2fb65640 1 -- start start 2024-09-21T03:28:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.304+0000 7f3e2fb65640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e28072d70 0x7f3e2809b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.304+0000 7f3e2fb65640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3e2812fc70 0x7f3e2809b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.304+0000 7f3e2fb65640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3e281305c0 0x7f3e2809bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.304+0000 7f3e2fb65640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f3e28138890 con 0x7f3e28072d70 2024-09-21T03:28:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.304+0000 7f3e2fb65640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3e28138710 con 0x7f3e281305c0 2024-09-21T03:28:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.304+0000 7f3e2fb65640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3e28138a10 con 0x7f3e2812fc70 2024-09-21T03:28:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2d100640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3e2812fc70 0x7f3e2809b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.308 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2d901640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e28072d70 0x7f3e2809b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2d100640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3e2812fc70 0x7f3e2809b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:35610/0 (socket says 172.21.15.27:35610) 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2d100640 1 -- 172.21.15.27:0/1922036139 learned_addr learned my addr 172.21.15.27:0/1922036139 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2d901640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e28072d70 0x7f3e2809b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44232/0 (socket says 172.21.15.27:44232) 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2e102640 1 --2- 172.21.15.27:0/1922036139 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3e281305c0 0x7f3e2809bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2d100640 1 -- 172.21.15.27:0/1922036139 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3e281305c0 msgr2=0x7f3e2809bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2d100640 1 --2- 172.21.15.27:0/1922036139 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3e281305c0 0x7f3e2809bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2d100640 1 -- 172.21.15.27:0/1922036139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e28072d70 msgr2=0x7f3e2809b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2d100640 1 --2- 172.21.15.27:0/1922036139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e28072d70 0x7f3e2809b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2d100640 1 -- 172.21.15.27:0/1922036139 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e2812be80 con 0x7f3e2812fc70 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2d100640 1 --2- 172.21.15.27:0/1922036139 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3e2812fc70 0x7f3e2809b9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3e18000c00 tx=0x7f3e180038a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e1e7fc640 1 -- 172.21.15.27:0/1922036139 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1803e030 con 0x7f3e2812fc70 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e1e7fc640 1 -- 172.21.15.27:0/1922036139 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3e18003db0 con 0x7f3e2812fc70 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2fb65640 1 -- 172.21.15.27:0/1922036139 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e2812c6c0 con 0x7f3e2812fc70 2024-09-21T03:28:53.309 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e1e7fc640 1 -- 172.21.15.27:0/1922036139 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e18035040 con 0x7f3e2812fc70 2024-09-21T03:28:53.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.305+0000 7f3e2fb65640 1 -- 172.21.15.27:0/1922036139 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e28129fd0 con 0x7f3e2812fc70 2024-09-21T03:28:53.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.306+0000 7f3e2fb65640 1 -- 172.21.15.27:0/1922036139 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e280729d0 con 0x7f3e2812fc70 2024-09-21T03:28:53.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.307+0000 7f3e1e7fc640 1 -- 172.21.15.27:0/1922036139 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f3e18041030 con 0x7f3e2812fc70 2024-09-21T03:28:53.310 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.307+0000 7f3e1e7fc640 1 -- 172.21.15.27:0/1922036139 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f3e180896f0 con 0x7f3e2812fc70 2024-09-21T03:28:53.313 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.311+0000 7f3e1e7fc640 1 -- 172.21.15.27:0/1922036139 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f3e280729d0 con 0x7f3e2812fc70 2024-09-21T03:28:53.467 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.465+0000 7f3e2fb65640 1 -- 172.21.15.27:0/1922036139 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3e2812cd10 con 0x7f3e0c0514b0 2024-09-21T03:28:53.468 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.466+0000 7f3e1e7fc640 1 -- 172.21.15.27:0/1922036139 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+32004 (unknown 2965378022 0 729647667) 0x7f3e24009850 con 0x7f3e0c0514b0 2024-09-21T03:28:53.468 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:53.468 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-21T03:28:53.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.467+0000 7f3e2fb65640 1 -- 172.21.15.27:0/1922036139 >> v1:172.21.15.27:6800/2405211194 conn(0x7f3e0c0514b0 legacy=0x7f3e0c0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.467+0000 7f3e2fb65640 1 -- 172.21.15.27:0/1922036139 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3e2812fc70 msgr2=0x7f3e2809b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.467+0000 7f3e2fb65640 1 --2- 172.21.15.27:0/1922036139 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3e2812fc70 0x7f3e2809b9c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3e18000c00 tx=0x7f3e180038a0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.467+0000 7f3e2fb65640 1 -- 172.21.15.27:0/1922036139 shutdown_connections 2024-09-21T03:28:53.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.467+0000 7f3e2fb65640 1 --2- 172.21.15.27:0/1922036139 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3e281305c0 0x7f3e2809bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.467+0000 7f3e2fb65640 1 --2- 172.21.15.27:0/1922036139 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f3e2812fc70 0x7f3e2809b9c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.467+0000 7f3e2fb65640 1 --2- 172.21.15.27:0/1922036139 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e28072d70 0x7f3e2809b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.468+0000 7f3e2fb65640 1 -- 172.21.15.27:0/1922036139 >> 172.21.15.27:0/1922036139 conn(0x7f3e280a1f40 msgr2=0x7f3e28136630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:53.470 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.468+0000 7f3e2fb65640 1 -- 172.21.15.27:0/1922036139 shutdown_connections 2024-09-21T03:28:53.471 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.468+0000 7f3e2fb65640 1 -- 172.21.15.27:0/1922036139 wait complete. 2024-09-21T03:28:53.481 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-21T03:28:52.300195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":151,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":116608,"kb_used_data":2280,"kb_used_omap":17,"kb_used_meta":114222,"kb_avail":377370752,"statfs":{"total":386547056640,"available":386427650048,"internally_reserved":0,"allocated":2334720,"data_stored":1576729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17729,"internal_metadata":116964031},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001795"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982313+0000","last_change":"2024-09-21T03:28:44.676220+0000","last_active":"2024-09-21T03:28:47.982313+0000","last_peered":"2024-09-21T03:28:47.982313+0000","last_clean":"2024-09-21T03:28:47.982313+0000","last_became_active":"2024-09-21T03:28:44.675664+0000","last_became_peered":"2024-09-21T03:28:44.675664+0000","last_unstale":"2024-09-21T03:28:47.982313+0000","last_undegraded":"2024-09-21T03:28:47.982313+0000","last_fullsized":"2024-09-21T03:28:47.982313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:02.081362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393285,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249352+0000","last_change":"2024-09-21T03:28:40.835755+0000","last_active":"2024-09-21T03:28:49.249352+0000","last_peered":"2024-09-21T03:28:49.249352+0000","last_clean":"2024-09-21T03:28:49.249352+0000","last_became_active":"2024-09-21T03:28:40.835171+0000","last_became_peered":"2024-09-21T03:28:40.835171+0000","last_unstale":"2024-09-21T03:28:49.249352+0000","last_undegraded":"2024-09-21T03:28:49.249352+0000","last_fullsized":"2024-09-21T03:28:49.249352+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:17:17.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044937200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982197+0000","last_change":"2024-09-21T03:28:44.676192+0000","last_active":"2024-09-21T03:28:47.982197+0000","last_peered":"2024-09-21T03:28:47.982197+0000","last_clean":"2024-09-21T03:28:47.982197+0000","last_became_active":"2024-09-21T03:28:44.675680+0000","last_became_peered":"2024-09-21T03:28:44.675680+0000","last_unstale":"2024-09-21T03:28:47.982197+0000","last_undegraded":"2024-09-21T03:28:47.982197+0000","last_fullsized":"2024-09-21T03:28:47.982197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:21:10.791750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249310+0000","last_change":"2024-09-21T03:28:40.835586+0000","last_active":"2024-09-21T03:28:49.249310+0000","last_peered":"2024-09-21T03:28:49.249310+0000","last_clean":"2024-09-21T03:28:49.249310+0000","last_became_active":"2024-09-21T03:28:40.835243+0000","last_became_peered":"2024-09-21T03:28:40.835243+0000","last_unstale":"2024-09-21T03:28:49.249310+0000","last_undegraded":"2024-09-21T03:28:49.249310+0000","last_fullsized":"2024-09-21T03:28:49.249310+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:31.997034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049584300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":47,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982077+0000","last_change":"2024-09-21T03:28:43.852388+0000","last_active":"2024-09-21T03:28:47.982077+0000","last_peered":"2024-09-21T03:28:47.982077+0000","last_clean":"2024-09-21T03:28:47.982077+0000","last_became_active":"2024-09-21T03:28:43.851555+0000","last_became_peered":"2024-09-21T03:28:43.851555+0000","last_unstale":"2024-09-21T03:28:47.982077+0000","last_undegraded":"2024-09-21T03:28:47.982077+0000","last_fullsized":"2024-09-21T03:28:47.982077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:37:19.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010094349999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:48.996126+0000","last_change":"2024-09-21T03:28:43.850674+0000","last_active":"2024-09-21T03:28:48.996126+0000","last_peered":"2024-09-21T03:28:48.996126+0000","last_clean":"2024-09-21T03:28:48.996126+0000","last_became_active":"2024-09-21T03:28:43.850058+0000","last_became_peered":"2024-09-21T03:28:43.850058+0000","last_unstale":"2024-09-21T03:28:48.996126+0000","last_undegraded":"2024-09-21T03:28:48.996126+0000","last_fullsized":"2024-09-21T03:28:48.996126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:58:47.763461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056217599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.853213+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.851256+0000","last_became_peered":"2024-09-21T03:28:43.851256+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:03.534378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045280299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":46,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.850214+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.849169+0000","last_became_peered":"2024-09-21T03:28:43.849169+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:34.860226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'104","reported_seq":201,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249233+0000","last_change":"2024-09-21T03:28:40.835338+0000","last_active":"2024-09-21T03:28:49.249233+0000","last_peered":"2024-09-21T03:28:49.249233+0000","last_clean":"2024-09-21T03:28:49.249233+0000","last_became_active":"2024-09-21T03:28:40.834765+0000","last_became_peered":"2024-09-21T03:28:40.834765+0000","last_unstale":"2024-09-21T03:28:49.249233+0000","last_undegraded":"2024-09-21T03:28:49.249233+0000","last_fullsized":"2024-09-21T03:28:49.249233+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_clean_scrub_stamp":"2024-09-21T03:27:08.233475+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:57.980901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":31,"seq":133143986180,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":276,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":282624,"data_stored":96334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":28176047},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":29,"seq":124554051588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":280,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":286720,"data_stored":99880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":864,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":884736,"data_stored":690267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32435888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":27,"seq":115964116996,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":860,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":880640,"data_stored":690248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T03:28:53.481 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T03:28:53.482 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T03:28:53.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.594+0000 7f8d777ae640 1 -- 172.21.15.27:0/3953097977 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d700718e0 msgr2=0x7f8d7012c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.596 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.594+0000 7f8d777ae640 1 --2- 172.21.15.27:0/3953097977 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d700718e0 0x7f8d7012c4c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8d60005fd0 tx=0x7f8d6002a820 comp rx=0 tx=0).stop 2024-09-21T03:28:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.594+0000 7f8d777ae640 1 -- 172.21.15.27:0/3953097977 shutdown_connections 2024-09-21T03:28:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.594+0000 7f8d777ae640 1 --2- 172.21.15.27:0/3953097977 >> v2:172.21.15.27:3300/0 conn(0x7f8d7012fd00 0x7f8d70137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.594+0000 7f8d777ae640 1 --2- 172.21.15.27:0/3953097977 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d700718e0 0x7f8d7012c4c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.594+0000 7f8d777ae640 1 -- 172.21.15.27:0/3953097977 >> 172.21.15.27:0/3953097977 conn(0x7f8d700a1d70 msgr2=0x7f8d700a2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.594+0000 7f8d777ae640 1 -- 172.21.15.27:0/3953097977 shutdown_connections 2024-09-21T03:28:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.594+0000 7f8d777ae640 1 -- 172.21.15.27:0/3953097977 wait complete. 2024-09-21T03:28:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.595+0000 7f8d777ae640 1 Processor -- start 2024-09-21T03:28:53.597 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.595+0000 7f8d777ae640 1 -- start start 2024-09-21T03:28:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d777ae640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d700718e0 0x7f8d701c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d777ae640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7012ca00 0x7f8d701c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d777ae640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8d7012fd00 0x7f8d701c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d777ae640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8d701386d0 con 0x7f8d7012ca00 2024-09-21T03:28:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d777ae640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f8d70138550 con 0x7f8d700718e0 2024-09-21T03:28:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d777ae640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f8d70138850 con 0x7f8d7012fd00 2024-09-21T03:28:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d74d49640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7012ca00 0x7f8d701c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d7554a640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d700718e0 0x7f8d701c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.600 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d7554a640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d700718e0 0x7f8d701c5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:46260/0 (socket says 172.21.15.27:46260) 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d75d4b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8d7012fd00 0x7f8d701c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d74d49640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7012ca00 0x7f8d701c6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44248/0 (socket says 172.21.15.27:44248) 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d7554a640 1 -- 172.21.15.27:0/1777307185 learned_addr learned my addr 172.21.15.27:0/1777307185 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d75d4b640 1 -- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d700718e0 msgr2=0x7f8d701c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d75d4b640 1 --2- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d700718e0 0x7f8d701c5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d75d4b640 1 -- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7012ca00 msgr2=0x7f8d701c6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d75d4b640 1 --2- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7012ca00 0x7f8d701c6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d75d4b640 1 -- 172.21.15.27:0/1777307185 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d70134770 con 0x7f8d7012fd00 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d7554a640 1 --2- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d700718e0 0x7f8d701c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d74d49640 1 --2- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7012ca00 0x7f8d701c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.596+0000 7f8d75d4b640 1 --2- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8d7012fd00 0x7f8d701c65a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d6800e940 tx=0x7f8d6800ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.597+0000 7f8d6dffb640 1 -- 172.21.15.27:0/1777307185 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6800ccc0 con 0x7f8d7012fd00 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.597+0000 7f8d6dffb640 1 -- 172.21.15.27:0/1777307185 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8d68004020 con 0x7f8d7012fd00 2024-09-21T03:28:53.601 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.597+0000 7f8d777ae640 1 -- 172.21.15.27:0/1777307185 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d70133920 con 0x7f8d7012fd00 2024-09-21T03:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.597+0000 7f8d6dffb640 1 -- 172.21.15.27:0/1777307185 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6800ce60 con 0x7f8d7012fd00 2024-09-21T03:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.597+0000 7f8d777ae640 1 -- 172.21.15.27:0/1777307185 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d70131890 con 0x7f8d7012fd00 2024-09-21T03:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.598+0000 7f8d777ae640 1 -- 172.21.15.27:0/1777307185 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d7012c4c0 con 0x7f8d7012fd00 2024-09-21T03:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.598+0000 7f8d6dffb640 1 -- 172.21.15.27:0/1777307185 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f8d680035f0 con 0x7f8d7012fd00 2024-09-21T03:28:53.602 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.598+0000 7f8d6dffb640 1 -- 172.21.15.27:0/1777307185 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f8d680697d0 con 0x7f8d7012fd00 2024-09-21T03:28:53.604 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.602+0000 7f8d6dffb640 1 -- 172.21.15.27:0/1777307185 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f8d68068880 con 0x7f8d7012fd00 2024-09-21T03:28:53.761 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.759+0000 7f8d777ae640 1 -- 172.21.15.27:0/1777307185 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8d7006dda0 con 0x7f8d5c0514b0 2024-09-21T03:28:53.762 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.760+0000 7f8d6dffb640 1 -- 172.21.15.27:0/1777307185 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+32004 (unknown 2965378022 0 729647667) 0x7f8d7006dda0 con 0x7f8d5c0514b0 2024-09-21T03:28:53.762 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:53.763 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-21T03:28:53.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.762+0000 7f8d777ae640 1 -- 172.21.15.27:0/1777307185 >> v1:172.21.15.27:6800/2405211194 conn(0x7f8d5c0514b0 legacy=0x7f8d5c0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.762+0000 7f8d777ae640 1 -- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8d7012fd00 msgr2=0x7f8d701c65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.764 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.762+0000 7f8d777ae640 1 --2- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8d7012fd00 0x7f8d701c65a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8d6800e940 tx=0x7f8d6800ee00 comp rx=0 tx=0).stop 2024-09-21T03:28:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.762+0000 7f8d777ae640 1 -- 172.21.15.27:0/1777307185 shutdown_connections 2024-09-21T03:28:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.762+0000 7f8d777ae640 1 --2- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8d7012fd00 0x7f8d701c65a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.762+0000 7f8d777ae640 1 --2- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8d7012ca00 0x7f8d701c6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.762+0000 7f8d777ae640 1 --2- 172.21.15.27:0/1777307185 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f8d700718e0 0x7f8d701c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.762+0000 7f8d777ae640 1 -- 172.21.15.27:0/1777307185 >> 172.21.15.27:0/1777307185 conn(0x7f8d700a1d70 msgr2=0x7f8d7012f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.762+0000 7f8d777ae640 1 -- 172.21.15.27:0/1777307185 shutdown_connections 2024-09-21T03:28:53.765 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.762+0000 7f8d777ae640 1 -- 172.21.15.27:0/1777307185 wait complete. 2024-09-21T03:28:53.775 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-21T03:28:52.300195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":151,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":116608,"kb_used_data":2280,"kb_used_omap":17,"kb_used_meta":114222,"kb_avail":377370752,"statfs":{"total":386547056640,"available":386427650048,"internally_reserved":0,"allocated":2334720,"data_stored":1576729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17729,"internal_metadata":116964031},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001795"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982313+0000","last_change":"2024-09-21T03:28:44.676220+0000","last_active":"2024-09-21T03:28:47.982313+0000","last_peered":"2024-09-21T03:28:47.982313+0000","last_clean":"2024-09-21T03:28:47.982313+0000","last_became_active":"2024-09-21T03:28:44.675664+0000","last_became_peered":"2024-09-21T03:28:44.675664+0000","last_unstale":"2024-09-21T03:28:47.982313+0000","last_undegraded":"2024-09-21T03:28:47.982313+0000","last_fullsized":"2024-09-21T03:28:47.982313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:02.081362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393285,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249352+0000","last_change":"2024-09-21T03:28:40.835755+0000","last_active":"2024-09-21T03:28:49.249352+0000","last_peered":"2024-09-21T03:28:49.249352+0000","last_clean":"2024-09-21T03:28:49.249352+0000","last_became_active":"2024-09-21T03:28:40.835171+0000","last_became_peered":"2024-09-21T03:28:40.835171+0000","last_unstale":"2024-09-21T03:28:49.249352+0000","last_undegraded":"2024-09-21T03:28:49.249352+0000","last_fullsized":"2024-09-21T03:28:49.249352+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:17:17.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044937200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982197+0000","last_change":"2024-09-21T03:28:44.676192+0000","last_active":"2024-09-21T03:28:47.982197+0000","last_peered":"2024-09-21T03:28:47.982197+0000","last_clean":"2024-09-21T03:28:47.982197+0000","last_became_active":"2024-09-21T03:28:44.675680+0000","last_became_peered":"2024-09-21T03:28:44.675680+0000","last_unstale":"2024-09-21T03:28:47.982197+0000","last_undegraded":"2024-09-21T03:28:47.982197+0000","last_fullsized":"2024-09-21T03:28:47.982197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:21:10.791750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249310+0000","last_change":"2024-09-21T03:28:40.835586+0000","last_active":"2024-09-21T03:28:49.249310+0000","last_peered":"2024-09-21T03:28:49.249310+0000","last_clean":"2024-09-21T03:28:49.249310+0000","last_became_active":"2024-09-21T03:28:40.835243+0000","last_became_peered":"2024-09-21T03:28:40.835243+0000","last_unstale":"2024-09-21T03:28:49.249310+0000","last_undegraded":"2024-09-21T03:28:49.249310+0000","last_fullsized":"2024-09-21T03:28:49.249310+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:31.997034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049584300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":47,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982077+0000","last_change":"2024-09-21T03:28:43.852388+0000","last_active":"2024-09-21T03:28:47.982077+0000","last_peered":"2024-09-21T03:28:47.982077+0000","last_clean":"2024-09-21T03:28:47.982077+0000","last_became_active":"2024-09-21T03:28:43.851555+0000","last_became_peered":"2024-09-21T03:28:43.851555+0000","last_unstale":"2024-09-21T03:28:47.982077+0000","last_undegraded":"2024-09-21T03:28:47.982077+0000","last_fullsized":"2024-09-21T03:28:47.982077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:37:19.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010094349999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:48.996126+0000","last_change":"2024-09-21T03:28:43.850674+0000","last_active":"2024-09-21T03:28:48.996126+0000","last_peered":"2024-09-21T03:28:48.996126+0000","last_clean":"2024-09-21T03:28:48.996126+0000","last_became_active":"2024-09-21T03:28:43.850058+0000","last_became_peered":"2024-09-21T03:28:43.850058+0000","last_unstale":"2024-09-21T03:28:48.996126+0000","last_undegraded":"2024-09-21T03:28:48.996126+0000","last_fullsized":"2024-09-21T03:28:48.996126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:58:47.763461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056217599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.853213+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.851256+0000","last_became_peered":"2024-09-21T03:28:43.851256+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:03.534378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045280299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":46,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.850214+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.849169+0000","last_became_peered":"2024-09-21T03:28:43.849169+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:34.860226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'104","reported_seq":201,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249233+0000","last_change":"2024-09-21T03:28:40.835338+0000","last_active":"2024-09-21T03:28:49.249233+0000","last_peered":"2024-09-21T03:28:49.249233+0000","last_clean":"2024-09-21T03:28:49.249233+0000","last_became_active":"2024-09-21T03:28:40.834765+0000","last_became_peered":"2024-09-21T03:28:40.834765+0000","last_unstale":"2024-09-21T03:28:49.249233+0000","last_undegraded":"2024-09-21T03:28:49.249233+0000","last_fullsized":"2024-09-21T03:28:49.249233+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_clean_scrub_stamp":"2024-09-21T03:27:08.233475+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:57.980901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":31,"seq":133143986180,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":276,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":282624,"data_stored":96334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":28176047},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":29,"seq":124554051588,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":280,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":286720,"data_stored":99880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":864,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":884736,"data_stored":690267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32435888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":27,"seq":115964116996,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":860,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":880640,"data_stored":690248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T03:28:53.776 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T03:28:53.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.888+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/1160333870 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7cd0071a30 msgr2=0x7f7cd0070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.890 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.888+0000 7f7cd7ea5640 1 --2- 172.21.15.27:0/1160333870 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7cd0071a30 0x7f7cd0070cb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009830 tx=0x7f7ccc02a3f0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.888+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/1160333870 shutdown_connections 2024-09-21T03:28:53.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.888+0000 7f7cd7ea5640 1 --2- 172.21.15.27:0/1160333870 >> v2:172.21.15.27:3300/0 conn(0x7f7cd0130280 0x7f7cd012c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.888+0000 7f7cd7ea5640 1 --2- 172.21.15.27:0/1160333870 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7cd0071a30 0x7f7cd0070cb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.888+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/1160333870 >> 172.21.15.27:0/1160333870 conn(0x7f7cd00a2050 msgr2=0x7f7cd00a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:53.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.888+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/1160333870 shutdown_connections 2024-09-21T03:28:53.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.888+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/1160333870 wait complete. 2024-09-21T03:28:53.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.889+0000 7f7cd7ea5640 1 Processor -- start 2024-09-21T03:28:53.891 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.889+0000 7f7cd7ea5640 1 -- start start 2024-09-21T03:28:53.893 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd7ea5640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7cd0071a30 0x7f7cd009b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd7ea5640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7cd012f900 0x7f7cd009b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd7ea5640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd0130280 0x7f7cd009bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd7ea5640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd0138490 con 0x7f7cd0130280 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd7ea5640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f7cd0138310 con 0x7f7cd0071a30 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd7ea5640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f7cd0138610 con 0x7f7cd012f900 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd5c41640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7cd0071a30 0x7f7cd009b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd5c41640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7cd0071a30 0x7f7cd009b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:46272/0 (socket says 172.21.15.27:46272) 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd5c41640 1 -- 172.21.15.27:0/3019195679 learned_addr learned my addr 172.21.15.27:0/3019195679 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd6442640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd0130280 0x7f7cd009bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd5440640 1 --2- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7cd012f900 0x7f7cd009b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd5c41640 1 -- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7cd012f900 msgr2=0x7f7cd009b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.894 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd5c41640 1 --2- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7cd012f900 0x7f7cd009b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd5c41640 1 -- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd0130280 msgr2=0x7f7cd009bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:53.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd5c41640 1 --2- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd0130280 0x7f7cd009bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:53.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd5c41640 1 -- 172.21.15.27:0/3019195679 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd0134810 con 0x7f7cd0071a30 2024-09-21T03:28:53.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd6442640 1 --2- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd0130280 0x7f7cd009bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:53.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cd5c41640 1 --2- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7cd0071a30 0x7f7cd009b480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc000c00 tx=0x7f7ccc033c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:53.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.890+0000 7f7cc27fc640 1 -- 172.21.15.27:0/3019195679 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc031d60 con 0x7f7cd0071a30 2024-09-21T03:28:53.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.891+0000 7f7cc27fc640 1 -- 172.21.15.27:0/3019195679 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f7ccc033e10 con 0x7f7cd0071a30 2024-09-21T03:28:53.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.891+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/3019195679 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd01339c0 con 0x7f7cd0071a30 2024-09-21T03:28:53.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.891+0000 7f7cc27fc640 1 -- 172.21.15.27:0/3019195679 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc0332b0 con 0x7f7cd0071a30 2024-09-21T03:28:53.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.891+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/3019195679 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd0131780 con 0x7f7cd0071a30 2024-09-21T03:28:53.895 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.891+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/3019195679 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd0070cb0 con 0x7f7cd0071a30 2024-09-21T03:28:53.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.892+0000 7f7cc27fc640 1 -- 172.21.15.27:0/3019195679 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f7ccc033450 con 0x7f7cd0071a30 2024-09-21T03:28:53.896 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.892+0000 7f7cc27fc640 1 -- 172.21.15.27:0/3019195679 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f7ccc08a2a0 con 0x7f7cd0071a30 2024-09-21T03:28:53.898 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:53.896+0000 7f7cc27fc640 1 -- 172.21.15.27:0/3019195679 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f7ccc03f030 con 0x7f7cd0071a30 2024-09-21T03:28:54.058 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.056+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/3019195679 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7cd006dda0 con 0x7f7cd0071a30 2024-09-21T03:28:54.059 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.057+0000 7f7cc27fc640 1 -- 172.21.15.27:0/3019195679 <== mon.1 v2:172.21.15.27:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+9171 (secure 0 0 0) 0x7f7ccc089350 con 0x7f7cd0071a30 2024-09-21T03:28:54.059 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:54.059 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"fsid":"22a90953-a6e7-4790-ab76-aa25cec033b2","created":"2024-09-21T03:26:54.150622+0000","modified":"2024-09-21T03:28:48.984591+0000","last_up_change":"2024-09-21T03:28:42.840845+0000","last_in_change":"2024-09-21T03:26:56.875414+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"squid","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T03:27:07.545484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T03:27:11.408521+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5bfee00c-9ea5-41bd-b128-5eb75667f4f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":21,"up_from":27,"up_thru":27,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6801","nonce":1914045614}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6802","nonce":1914045614}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6804","nonce":1914045614}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6803","nonce":1914045614}]},"public_addr":"172.21.15.27:6801/1914045614","cluster_addr":"172.21.15.27:6802/1914045614","heartbeat_back_addr":"172.21.15.27:6804/1914045614","heartbeat_front_addr":"172.21.15.27:6803/1914045614","state":["exists","up"]},{"osd":1,"uuid":"121ea55e-e6d7-4d7e-bbc8-ef3fac12060f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":22,"up_from":28,"up_thru":28,"down_at":23,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6805","nonce":580113737}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6806","nonce":580113737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6808","nonce":580113737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6807","nonce":580113737}]},"public_addr":"172.21.15.27:6805/580113737","cluster_addr":"172.21.15.27:6806/580113737","heartbeat_back_addr":"172.21.15.27:6808/580113737","heartbeat_front_addr":"172.21.15.27:6807/580113737","state":["exists","up"]},{"osd":2,"uuid":"b0834a51-8a1a-4bdf-9e82-9b77d445c2a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":23,"up_from":29,"up_thru":31,"down_at":24,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6809","nonce":2016211836}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6810","nonce":2016211836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6812","nonce":2016211836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6811","nonce":2016211836}]},"public_addr":"172.21.15.27:6809/2016211836","cluster_addr":"172.21.15.27:6810/2016211836","heartbeat_back_addr":"172.21.15.27:6812/2016211836","heartbeat_front_addr":"172.21.15.27:6811/2016211836","state":["exists","up"]},{"osd":3,"uuid":"344f8d01-27ba-4598-a138-5fdde9141b60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":24,"up_from":31,"up_thru":31,"down_at":25,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6813","nonce":1188450781}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6814","nonce":1188450781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6816","nonce":1188450781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.27:6815","nonce":1188450781}]},"public_addr":"172.21.15.27:6813/1188450781","cluster_addr":"172.21.15.27:6814/1188450781","heartbeat_back_addr":"172.21.15.27:6816/1188450781","heartbeat_front_addr":"172.21.15.27:6815/1188450781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.620971+0000","dead_epoch":22},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.459037+0000","dead_epoch":23},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:03.899158+0000","dead_epoch":24},{"osd":3,"down_stamp":"2024-09-21T03:28:35.686744+0000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T03:27:04.132502+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.27:0/1436650201":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3459873364":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/3140689679":"2024-09-22T03:28:28.273799+0000","172.21.15.27:6800/1098903051":"2024-09-22T03:28:28.273799+0000","172.21.15.27:0/1012963840":"2024-09-22T03:28:28.273799+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T03:28:54.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.058+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/3019195679 >> v1:172.21.15.27:6800/2405211194 conn(0x7f7cb40514b0 legacy=0x7f7cb40539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:54.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.058+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7cd0071a30 msgr2=0x7f7cd009b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:54.061 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.058+0000 7f7cd7ea5640 1 --2- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7cd0071a30 0x7f7cd009b480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc000c00 tx=0x7f7ccc033c20 comp rx=0 tx=0).stop 2024-09-21T03:28:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.059+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/3019195679 shutdown_connections 2024-09-21T03:28:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.059+0000 7f7cd7ea5640 1 --2- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7cd0130280 0x7f7cd009bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.059+0000 7f7cd7ea5640 1 --2- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f7cd012f900 0x7f7cd009b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.059+0000 7f7cd7ea5640 1 --2- 172.21.15.27:0/3019195679 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f7cd0071a30 0x7f7cd009b480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.059+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/3019195679 >> 172.21.15.27:0/3019195679 conn(0x7f7cd00a2050 msgr2=0x7f7cd0073e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.059+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/3019195679 shutdown_connections 2024-09-21T03:28:54.062 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:54.059+0000 7f7cd7ea5640 1 -- 172.21.15.27:0/3019195679 wait complete. 2024-09-21T03:28:55.072 INFO:tasks.ceph:Scrubbing osd.0 2024-09-21T03:28:55.073 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-21T03:28:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.208+0000 7ffac7c61640 1 -- 172.21.15.27:0/3732819599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac006cd90 msgr2=0x7ffac0072cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.208+0000 7ffac7c61640 1 --2- 172.21.15.27:0/3732819599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac006cd90 0x7ffac0072cb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009830 tx=0x7ffabc02a3f0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.208+0000 7ffac7c61640 1 -- 172.21.15.27:0/3732819599 shutdown_connections 2024-09-21T03:28:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.208+0000 7ffac7c61640 1 --2- 172.21.15.27:0/3732819599 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ffac0097070 0x7ffac0097480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.208+0000 7ffac7c61640 1 --2- 172.21.15.27:0/3732819599 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ffac00731f0 0x7ffac0073600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.208+0000 7ffac7c61640 1 --2- 172.21.15.27:0/3732819599 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac006cd90 0x7ffac0072cb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.211 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.208+0000 7ffac7c61640 1 -- 172.21.15.27:0/3732819599 >> 172.21.15.27:0/3732819599 conn(0x7ffac009bcb0 msgr2=0x7ffac009c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.208+0000 7ffac7c61640 1 -- 172.21.15.27:0/3732819599 shutdown_connections 2024-09-21T03:28:55.212 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.208+0000 7ffac7c61640 1 -- 172.21.15.27:0/3732819599 wait complete. 2024-09-21T03:28:55.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.209+0000 7ffac7c61640 1 Processor -- start 2024-09-21T03:28:55.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.209+0000 7ffac7c61640 1 -- start start 2024-09-21T03:28:55.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac7c61640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ffac006cd90 0x7ffac0122a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac7c61640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac00731f0 0x7ffac0122f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac7c61640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ffac0097070 0x7ffac01234d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac7c61640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0134210 con 0x7ffac00731f0 2024-09-21T03:28:55.214 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac7c61640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7ffac0134090 con 0x7ffac006cd90 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac7c61640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7ffac0134390 con 0x7ffac0097070 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac51fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac00731f0 0x7ffac0122f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac61fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ffac0097070 0x7ffac01234d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac51fc640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac00731f0 0x7ffac0122f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44274/0 (socket says 172.21.15.27:44274) 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac59fd640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ffac006cd90 0x7ffac0122a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac51fc640 1 -- 172.21.15.27:0/1141417127 learned_addr learned my addr 172.21.15.27:0/1141417127 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.210+0000 7ffac61fe640 1 --2- 172.21.15.27:0/1141417127 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ffac0097070 0x7ffac01234d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:35636/0 (socket says 172.21.15.27:35636) 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffac61fe640 1 -- 172.21.15.27:0/1141417127 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ffac006cd90 msgr2=0x7ffac0122a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffac61fe640 1 --2- 172.21.15.27:0/1141417127 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ffac006cd90 0x7ffac0122a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffac61fe640 1 -- 172.21.15.27:0/1141417127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac00731f0 msgr2=0x7ffac0122f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffac61fe640 1 --2- 172.21.15.27:0/1141417127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac00731f0 0x7ffac0122f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.215 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffac61fe640 1 -- 172.21.15.27:0/1141417127 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac0123bd0 con 0x7ffac0097070 2024-09-21T03:28:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffac59fd640 1 --2- 172.21.15.27:0/1141417127 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7ffac006cd90 0x7ffac0122a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffac51fc640 1 --2- 172.21.15.27:0/1141417127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ffac00731f0 0x7ffac0122f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffac61fe640 1 --2- 172.21.15.27:0/1141417127 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ffac0097070 0x7ffac01234d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ffab8005fd0 tx=0x7ffab800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffab67fc640 1 -- 172.21.15.27:0/1141417127 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab800cd80 con 0x7ffac0097070 2024-09-21T03:28:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffab67fc640 1 -- 172.21.15.27:0/1141417127 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7ffab8004c10 con 0x7ffac0097070 2024-09-21T03:28:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffac7c61640 1 -- 172.21.15.27:0/1141417127 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac0124420 con 0x7ffac0097070 2024-09-21T03:28:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffab67fc640 1 -- 172.21.15.27:0/1141417127 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab8005020 con 0x7ffac0097070 2024-09-21T03:28:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.211+0000 7ffac7c61640 1 -- 172.21.15.27:0/1141417127 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac012ea30 con 0x7ffac0097070 2024-09-21T03:28:55.216 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.212+0000 7ffac7c61640 1 -- 172.21.15.27:0/1141417127 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffac0073600 con 0x7ffac0097070 2024-09-21T03:28:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.212+0000 7ffab67fc640 1 -- 172.21.15.27:0/1141417127 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7ffab80051c0 con 0x7ffac0097070 2024-09-21T03:28:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.213+0000 7ffab67fc640 1 -- 172.21.15.27:0/1141417127 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7ffab8069c40 con 0x7ffac0097070 2024-09-21T03:28:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.213+0000 7ffab67fc640 1 -- 172.21.15.27:0/1141417127 --> v1:172.21.15.27:6801/1914045614 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffaa405ba80 con 0x7ffaa4057d80 2024-09-21T03:28:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.213+0000 7ffab67fc640 1 -- 172.21.15.27:0/1141417127 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ffab806c340 con 0x7ffac0097070 2024-09-21T03:28:55.217 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.214+0000 7ffab67fc640 1 -- 172.21.15.27:0/1141417127 <== osd.0 v1:172.21.15.27:6801/1914045614 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ffaa405ba80 con 0x7ffaa4057d80 2024-09-21T03:28:55.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.227+0000 7ffac7c61640 1 -- 172.21.15.27:0/1141417127 --> v1:172.21.15.27:6801/1914045614 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ffac0132990 con 0x7ffaa4057d80 2024-09-21T03:28:55.229 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.227+0000 7ffab67fc640 1 -- 172.21.15.27:0/1141417127 <== osd.0 v1:172.21.15.27:6801/1914045614 2 ==== command_reply(tid 2: 0 ) ==== 8+0+565 (unknown 0 0 125856537) 0x7ffac0132990 con 0x7ffaa4057d80 2024-09-21T03:28:55.229 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-21T03:28:55.229 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-21T03:28:55.229 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-21T03:28:55.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.227+0000 7ffac7c61640 1 -- 172.21.15.27:0/1141417127 >> v1:172.21.15.27:6801/1914045614 conn(0x7ffaa4057d80 legacy=0x7ffaa405a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.230 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.227+0000 7ffac7c61640 1 -- 172.21.15.27:0/1141417127 >> v1:172.21.15.27:6800/2405211194 conn(0x7ffaa40514b0 legacy=0x7ffaa40539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.227+0000 7ffac7c61640 1 -- 172.21.15.27:0/1141417127 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ffac0097070 msgr2=0x7ffac01234d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.227+0000 7ffac7c61640 1 --2- 172.21.15.27:0/1141417127 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7ffac0097070 0x7ffac01234d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ffab8005fd0 tx=0x7ffab800ee40 comp rx=0 tx=0).stop 2024-09-21T03:28:55.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.227+0000 7ffac61fe640 1 -- 172.21.15.27:0/1141417127 reap_dead start 2024-09-21T03:28:55.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.227+0000 7ffac7c61640 1 -- 172.21.15.27:0/1141417127 shutdown_connections 2024-09-21T03:28:55.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.227+0000 7ffac7c61640 1 -- 172.21.15.27:0/1141417127 >> 172.21.15.27:0/1141417127 conn(0x7ffac009bcb0 msgr2=0x7ffac0132030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:55.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.227+0000 7ffac7c61640 1 -- 172.21.15.27:0/1141417127 shutdown_connections 2024-09-21T03:28:55.231 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.227+0000 7ffac7c61640 1 -- 172.21.15.27:0/1141417127 wait complete. 2024-09-21T03:28:55.241 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-21T03:28:55.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.354+0000 7f273fb17640 1 -- 172.21.15.27:0/1504632211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738070ec0 msgr2=0x7f27381348b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.354+0000 7f273fb17640 1 --2- 172.21.15.27:0/1504632211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738070ec0 0x7f27381348b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2728009830 tx=0x7f272802a3f0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.354+0000 7f273fb17640 1 -- 172.21.15.27:0/1504632211 shutdown_connections 2024-09-21T03:28:55.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.354+0000 7f273fb17640 1 --2- 172.21.15.27:0/1504632211 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2738134df0 0x7f2738137210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.354+0000 7f273fb17640 1 --2- 172.21.15.27:0/1504632211 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738070ec0 0x7f27381348b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.354+0000 7f273fb17640 1 -- 172.21.15.27:0/1504632211 >> 172.21.15.27:0/1504632211 conn(0x7f27380a2050 msgr2=0x7f27380a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:55.357 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.354+0000 7f273fb17640 1 -- 172.21.15.27:0/1504632211 shutdown_connections 2024-09-21T03:28:55.358 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.354+0000 7f273fb17640 1 -- 172.21.15.27:0/1504632211 wait complete. 2024-09-21T03:28:55.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.355+0000 7f273fb17640 1 Processor -- start 2024-09-21T03:28:55.359 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273fb17640 1 -- start start 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273fb17640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738070ec0 0x7f27381c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273fb17640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f273812f630 0x7f27381c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273fb17640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2738134df0 0x7f27381c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273fb17640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f2738138570 con 0x7f2738070ec0 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273fb17640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f27381383f0 con 0x7f2738134df0 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273fb17640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f27381386f0 con 0x7f273812f630 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273d8b3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738070ec0 0x7f27381c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273d0b2640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f273812f630 0x7f27381c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273e0b4640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2738134df0 0x7f27381c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273d8b3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738070ec0 0x7f27381c59c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44286/0 (socket says 172.21.15.27:44286) 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.356+0000 7f273d8b3640 1 -- 172.21.15.27:0/2766910207 learned_addr learned my addr 172.21.15.27:0/2766910207 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f273d8b3640 1 -- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f273812f630 msgr2=0x7f27381c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.360 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f273d8b3640 1 --2- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f273812f630 0x7f27381c5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f273d8b3640 1 -- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2738134df0 msgr2=0x7f27381c6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f273d8b3640 1 --2- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2738134df0 0x7f27381c6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f273d8b3640 1 -- 172.21.15.27:0/2766910207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273809e8d0 con 0x7f2738070ec0 2024-09-21T03:28:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f273e0b4640 1 --2- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2738134df0 0x7f27381c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f273d0b2640 1 --2- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f273812f630 0x7f27381c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f273d8b3640 1 --2- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738070ec0 0x7f27381c59c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f273400c430 tx=0x7f273400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f272e7fc640 1 -- 172.21.15.27:0/2766910207 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734017070 con 0x7f2738070ec0 2024-09-21T03:28:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f272e7fc640 1 -- 172.21.15.27:0/2766910207 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f27340035f0 con 0x7f2738070ec0 2024-09-21T03:28:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f272e7fc640 1 -- 172.21.15.27:0/2766910207 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734003a40 con 0x7f2738070ec0 2024-09-21T03:28:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f273fb17640 1 -- 172.21.15.27:0/2766910207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2738130660 con 0x7f2738070ec0 2024-09-21T03:28:55.361 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.357+0000 7f273fb17640 1 -- 172.21.15.27:0/2766910207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f273809d530 con 0x7f2738070ec0 2024-09-21T03:28:55.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.358+0000 7f273fb17640 1 -- 172.21.15.27:0/2766910207 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f273812fb60 con 0x7f2738070ec0 2024-09-21T03:28:55.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.358+0000 7f272e7fc640 1 -- 172.21.15.27:0/2766910207 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f273401a070 con 0x7f2738070ec0 2024-09-21T03:28:55.362 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.359+0000 7f272e7fc640 1 -- 172.21.15.27:0/2766910207 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f2734068310 con 0x7f2738070ec0 2024-09-21T03:28:55.364 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.362+0000 7f272e7fc640 1 -- 172.21.15.27:0/2766910207 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f27340673c0 con 0x7f2738070ec0 2024-09-21T03:28:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.521+0000 7f273fb17640 1 -- 172.21.15.27:0/2766910207 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f273806dda0 con 0x7f271c0514b0 2024-09-21T03:28:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.522+0000 7f272e7fc640 1 -- 172.21.15.27:0/2766910207 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7f273806dda0 con 0x7f271c0514b0 2024-09-21T03:28:55.524 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 0 to deep-scrub 2024-09-21T03:28:55.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.524+0000 7f273fb17640 1 -- 172.21.15.27:0/2766910207 >> v1:172.21.15.27:6800/2405211194 conn(0x7f271c0514b0 legacy=0x7f271c0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.524+0000 7f273fb17640 1 -- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738070ec0 msgr2=0x7f27381c59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.526 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.524+0000 7f273fb17640 1 --2- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738070ec0 0x7f27381c59c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f273400c430 tx=0x7f273400c8f0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.524+0000 7f273fb17640 1 -- 172.21.15.27:0/2766910207 shutdown_connections 2024-09-21T03:28:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.524+0000 7f273fb17640 1 --2- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f2738134df0 0x7f27381c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.524+0000 7f273fb17640 1 --2- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f273812f630 0x7f27381c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.524+0000 7f273fb17640 1 --2- 172.21.15.27:0/2766910207 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2738070ec0 0x7f27381c59c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.524+0000 7f273fb17640 1 -- 172.21.15.27:0/2766910207 >> 172.21.15.27:0/2766910207 conn(0x7f27380a2050 msgr2=0x7f2738073760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.524+0000 7f273fb17640 1 -- 172.21.15.27:0/2766910207 shutdown_connections 2024-09-21T03:28:55.527 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.524+0000 7f273fb17640 1 -- 172.21.15.27:0/2766910207 wait complete. 2024-09-21T03:28:55.537 INFO:tasks.ceph:Scrubbing osd.1 2024-09-21T03:28:55.537 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-21T03:28:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.655+0000 7f6ed1556640 1 -- 172.21.15.27:0/1989721059 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6ecc070c20 msgr2=0x7f6ecc130250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.655+0000 7f6ed1556640 1 --2- 172.21.15.27:0/1989721059 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6ecc070c20 0x7f6ecc130250 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4005fd0 tx=0x7f6eb402a4a0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.657 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.655+0000 7f6ed1556640 1 -- 172.21.15.27:0/1989721059 shutdown_connections 2024-09-21T03:28:55.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.655+0000 7f6ed1556640 1 --2- 172.21.15.27:0/1989721059 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6ecc12c280 0x7f6ecc12c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.655+0000 7f6ed1556640 1 --2- 172.21.15.27:0/1989721059 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6ecc070c20 0x7f6ecc130250 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.655+0000 7f6ed1556640 1 --2- 172.21.15.27:0/1989721059 >> v2:172.21.15.27:3300/0 conn(0x7f6ecc12f7e0 0x7f6ecc12fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.655+0000 7f6ed1556640 1 -- 172.21.15.27:0/1989721059 >> 172.21.15.27:0/1989721059 conn(0x7f6ecc0a1f80 msgr2=0x7f6ecc0a2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:55.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.655+0000 7f6ed1556640 1 -- 172.21.15.27:0/1989721059 shutdown_connections 2024-09-21T03:28:55.658 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.655+0000 7f6ed1556640 1 -- 172.21.15.27:0/1989721059 wait complete. 2024-09-21T03:28:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.656+0000 7f6ed1556640 1 Processor -- start 2024-09-21T03:28:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.656+0000 7f6ed1556640 1 -- start start 2024-09-21T03:28:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ed1556640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6ecc070c20 0x7f6ecc09b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ed1556640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ecc12c280 0x7f6ecc09b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ed1556640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6ecc12f7e0 0x7f6ecc09bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ed1556640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f6ecc137540 con 0x7f6ecc12c280 2024-09-21T03:28:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ed1556640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f6ecc1373c0 con 0x7f6ecc070c20 2024-09-21T03:28:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ed1556640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f6ecc1376c0 con 0x7f6ecc12f7e0 2024-09-21T03:28:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ecaffd640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6ecc070c20 0x7f6ecc09b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.660 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ec3fff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ecc12c280 0x7f6ecc09b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ecaffd640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6ecc070c20 0x7f6ecc09b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:46310/0 (socket says 172.21.15.27:46310) 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ecaffd640 1 -- 172.21.15.27:0/661176933 learned_addr learned my addr 172.21.15.27:0/661176933 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ecb7fe640 1 --2- 172.21.15.27:0/661176933 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6ecc12f7e0 0x7f6ecc09bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ecaffd640 1 -- 172.21.15.27:0/661176933 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6ecc12f7e0 msgr2=0x7f6ecc09bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ecaffd640 1 --2- 172.21.15.27:0/661176933 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6ecc12f7e0 0x7f6ecc09bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ecaffd640 1 -- 172.21.15.27:0/661176933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ecc12c280 msgr2=0x7f6ecc09b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ecaffd640 1 --2- 172.21.15.27:0/661176933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6ecc12c280 0x7f6ecc09b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ecaffd640 1 -- 172.21.15.27:0/661176933 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ecc134410 con 0x7f6ecc070c20 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ecb7fe640 1 --2- 172.21.15.27:0/661176933 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f6ecc12f7e0 0x7f6ecc09bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ecaffd640 1 --2- 172.21.15.27:0/661176933 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6ecc070c20 0x7f6ecc09b480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00e790 tx=0x7f6ebc00eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ec37fe640 1 -- 172.21.15.27:0/661176933 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc00cab0 con 0x7f6ecc070c20 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ec37fe640 1 -- 172.21.15.27:0/661176933 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f6ebc00cd40 con 0x7f6ecc070c20 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ec37fe640 1 -- 172.21.15.27:0/661176933 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc010690 con 0x7f6ecc070c20 2024-09-21T03:28:55.661 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ed1556640 1 -- 172.21.15.27:0/661176933 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ecc135030 con 0x7f6ecc070c20 2024-09-21T03:28:55.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.657+0000 7f6ed1556640 1 -- 172.21.15.27:0/661176933 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ecc1355d0 con 0x7f6ecc070c20 2024-09-21T03:28:55.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.658+0000 7f6ed1556640 1 -- 172.21.15.27:0/661176933 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6ecc12c690 con 0x7f6ecc070c20 2024-09-21T03:28:55.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.659+0000 7f6ec37fe640 1 -- 172.21.15.27:0/661176933 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f6ebc010830 con 0x7f6ecc070c20 2024-09-21T03:28:55.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.659+0000 7f6ec37fe640 1 -- 172.21.15.27:0/661176933 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f6ebc0698d0 con 0x7f6ecc070c20 2024-09-21T03:28:55.662 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.659+0000 7f6ec37fe640 1 -- 172.21.15.27:0/661176933 --> v1:172.21.15.27:6805/580113737 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ebc025e50 con 0x7f6eac057d80 2024-09-21T03:28:55.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.659+0000 7f6ec37fe640 1 -- 172.21.15.27:0/661176933 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f6ebc06e2c0 con 0x7f6ecc070c20 2024-09-21T03:28:55.663 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.660+0000 7f6ec37fe640 1 -- 172.21.15.27:0/661176933 <== osd.1 v1:172.21.15.27:6805/580113737 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f6ebc025e50 con 0x7f6eac057d80 2024-09-21T03:28:55.675 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.673+0000 7f6ed1556640 1 -- 172.21.15.27:0/661176933 --> v1:172.21.15.27:6805/580113737 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6ecc135a80 con 0x7f6eac057d80 2024-09-21T03:28:55.676 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.673+0000 7f6ec37fe640 1 -- 172.21.15.27:0/661176933 <== osd.1 v1:172.21.15.27:6805/580113737 2 ==== command_reply(tid 2: 0 ) ==== 8+0+565 (unknown 0 0 125856537) 0x7f6ecc135a80 con 0x7f6eac057d80 2024-09-21T03:28:55.676 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-21T03:28:55.676 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-21T03:28:55.676 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-21T03:28:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.673+0000 7f6ed1556640 1 -- 172.21.15.27:0/661176933 >> v1:172.21.15.27:6805/580113737 conn(0x7f6eac057d80 legacy=0x7f6eac05a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.674+0000 7f6ed1556640 1 -- 172.21.15.27:0/661176933 >> v1:172.21.15.27:6800/2405211194 conn(0x7f6eac0514b0 legacy=0x7f6eac0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.674+0000 7f6ed1556640 1 -- 172.21.15.27:0/661176933 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6ecc070c20 msgr2=0x7f6ecc09b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.674+0000 7f6ed1556640 1 --2- 172.21.15.27:0/661176933 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f6ecc070c20 0x7f6ecc09b480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00e790 tx=0x7f6ebc00eb90 comp rx=0 tx=0).stop 2024-09-21T03:28:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.674+0000 7f6ecb7fe640 1 -- 172.21.15.27:0/661176933 reap_dead start 2024-09-21T03:28:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.674+0000 7f6ed1556640 1 -- 172.21.15.27:0/661176933 shutdown_connections 2024-09-21T03:28:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.674+0000 7f6ed1556640 1 -- 172.21.15.27:0/661176933 >> 172.21.15.27:0/661176933 conn(0x7f6ecc0a1f80 msgr2=0x7f6ecc12ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:55.677 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.674+0000 7f6ed1556640 1 -- 172.21.15.27:0/661176933 shutdown_connections 2024-09-21T03:28:55.678 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.674+0000 7f6ed1556640 1 -- 172.21.15.27:0/661176933 wait complete. 2024-09-21T03:28:55.687 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-21T03:28:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.800+0000 7f5e46881640 1 -- 172.21.15.27:0/508943884 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e4012f900 msgr2=0x7f5e401349b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.800+0000 7f5e46881640 1 --2- 172.21.15.27:0/508943884 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e4012f900 0x7f5e401349b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5e34005fd0 tx=0x7f5e3402a4b0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.800+0000 7f5e46881640 1 -- 172.21.15.27:0/508943884 shutdown_connections 2024-09-21T03:28:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.800+0000 7f5e46881640 1 --2- 172.21.15.27:0/508943884 >> v2:172.21.15.27:3300/0 conn(0x7f5e40130280 0x7f5e40136f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.800+0000 7f5e46881640 1 --2- 172.21.15.27:0/508943884 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e4012f900 0x7f5e401349b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.800+0000 7f5e46881640 1 -- 172.21.15.27:0/508943884 >> 172.21.15.27:0/508943884 conn(0x7f5e400a2050 msgr2=0x7f5e400a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.800+0000 7f5e46881640 1 -- 172.21.15.27:0/508943884 shutdown_connections 2024-09-21T03:28:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.800+0000 7f5e46881640 1 -- 172.21.15.27:0/508943884 wait complete. 2024-09-21T03:28:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.801+0000 7f5e46881640 1 Processor -- start 2024-09-21T03:28:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.801+0000 7f5e46881640 1 -- start start 2024-09-21T03:28:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e46881640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40071a30 0x7f5e4009b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.805 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e46881640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e4012f900 0x7f5e4009b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e46881640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e40130280 0x7f5e4009bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e46881640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f5e40138190 con 0x7f5e40071a30 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e46881640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f5e40138010 con 0x7f5e40130280 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e46881640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f5e40138920 con 0x7f5e4012f900 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e3ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40071a30 0x7f5e4009b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e3ffff640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40071a30 0x7f5e4009b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44328/0 (socket says 172.21.15.27:44328) 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e3ffff640 1 -- 172.21.15.27:0/2752202966 learned_addr learned my addr 172.21.15.27:0/2752202966 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e3f7fe640 1 --2- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e4012f900 0x7f5e4009b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e44e1e640 1 --2- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e40130280 0x7f5e4009bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e3ffff640 1 -- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e4012f900 msgr2=0x7f5e4009b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e3ffff640 1 --2- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e4012f900 0x7f5e4009b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e3ffff640 1 -- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e40130280 msgr2=0x7f5e4009bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e3ffff640 1 --2- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e40130280 0x7f5e4009bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e3ffff640 1 -- 172.21.15.27:0/2752202966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e401329a0 con 0x7f5e40071a30 2024-09-21T03:28:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.802+0000 7f5e3f7fe640 1 --2- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e4012f900 0x7f5e4009b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.803+0000 7f5e3ffff640 1 --2- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40071a30 0x7f5e4009b480 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000c3a0 tx=0x7f5e3000c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.803+0000 7f5e3cff9640 1 -- 172.21.15.27:0/2752202966 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30017070 con 0x7f5e40071a30 2024-09-21T03:28:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.803+0000 7f5e3cff9640 1 -- 172.21.15.27:0/2752202966 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f5e300053c0 con 0x7f5e40071a30 2024-09-21T03:28:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.803+0000 7f5e3cff9640 1 -- 172.21.15.27:0/2752202966 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30003850 con 0x7f5e40071a30 2024-09-21T03:28:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.803+0000 7f5e46881640 1 -- 172.21.15.27:0/2752202966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e401337f0 con 0x7f5e40071a30 2024-09-21T03:28:55.807 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.803+0000 7f5e46881640 1 -- 172.21.15.27:0/2752202966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e401307b0 con 0x7f5e40071a30 2024-09-21T03:28:55.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.804+0000 7f5e46881640 1 -- 172.21.15.27:0/2752202966 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e40070cb0 con 0x7f5e40071a30 2024-09-21T03:28:55.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.804+0000 7f5e3cff9640 1 -- 172.21.15.27:0/2752202966 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f5e3001a070 con 0x7f5e40071a30 2024-09-21T03:28:55.808 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.804+0000 7f5e3cff9640 1 -- 172.21.15.27:0/2752202966 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f5e300682d0 con 0x7f5e40071a30 2024-09-21T03:28:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.808+0000 7f5e3cff9640 1 -- 172.21.15.27:0/2752202966 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f5e30067380 con 0x7f5e40071a30 2024-09-21T03:28:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.970+0000 7f5e46881640 1 -- 172.21.15.27:0/2752202966 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f5e401320f0 con 0x7f5e2c0514b0 2024-09-21T03:28:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.971+0000 7f5e3cff9640 1 -- 172.21.15.27:0/2752202966 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7f5e401320f0 con 0x7f5e2c0514b0 2024-09-21T03:28:55.973 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 1 to deep-scrub 2024-09-21T03:28:55.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.973+0000 7f5e46881640 1 -- 172.21.15.27:0/2752202966 >> v1:172.21.15.27:6800/2405211194 conn(0x7f5e2c0514b0 legacy=0x7f5e2c0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.973+0000 7f5e46881640 1 -- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40071a30 msgr2=0x7f5e4009b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:55.975 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.973+0000 7f5e46881640 1 --2- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40071a30 0x7f5e4009b480 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5e3000c3a0 tx=0x7f5e3000c7a0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.973+0000 7f5e46881640 1 -- 172.21.15.27:0/2752202966 shutdown_connections 2024-09-21T03:28:55.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.973+0000 7f5e46881640 1 --2- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f5e40130280 0x7f5e4009bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.973+0000 7f5e46881640 1 --2- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f5e4012f900 0x7f5e4009b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.973+0000 7f5e46881640 1 --2- 172.21.15.27:0/2752202966 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5e40071a30 0x7f5e4009b480 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:55.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.973+0000 7f5e46881640 1 -- 172.21.15.27:0/2752202966 >> 172.21.15.27:0/2752202966 conn(0x7f5e400a2050 msgr2=0x7f5e401369b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:55.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.973+0000 7f5e46881640 1 -- 172.21.15.27:0/2752202966 shutdown_connections 2024-09-21T03:28:55.976 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:55.973+0000 7f5e46881640 1 -- 172.21.15.27:0/2752202966 wait complete. 2024-09-21T03:28:55.986 INFO:tasks.ceph:Scrubbing osd.2 2024-09-21T03:28:55.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-21T03:28:56.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.103+0000 7f9100868640 1 -- 172.21.15.27:0/1046471320 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f90fc129ab0 msgr2=0x7f90fc12c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.103+0000 7f9100868640 1 --2- 172.21.15.27:0/1046471320 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f90fc129ab0 0x7f90fc12c4b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009860 tx=0x7f90f002a460 comp rx=0 tx=0).stop 2024-09-21T03:28:56.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.103+0000 7f9100868640 1 -- 172.21.15.27:0/1046471320 shutdown_connections 2024-09-21T03:28:56.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.103+0000 7f9100868640 1 --2- 172.21.15.27:0/1046471320 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f90fc129ab0 0x7f90fc12c4b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.103+0000 7f9100868640 1 --2- 172.21.15.27:0/1046471320 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f90fc129160 0x7f90fc129570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.103+0000 7f9100868640 1 -- 172.21.15.27:0/1046471320 >> 172.21.15.27:0/1046471320 conn(0x7f90fc09bcb0 msgr2=0x7f90fc09c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:56.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.103+0000 7f9100868640 1 -- 172.21.15.27:0/1046471320 shutdown_connections 2024-09-21T03:28:56.106 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.103+0000 7f9100868640 1 -- 172.21.15.27:0/1046471320 wait complete. 2024-09-21T03:28:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.104+0000 7f9100868640 1 Processor -- start 2024-09-21T03:28:56.107 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f9100868640 1 -- start start 2024-09-21T03:28:56.108 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f9100868640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90fc129160 0x7f90fc097070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f9100868640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f90fc129ab0 0x7f90fc0975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f9100868640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f90fc12c9f0 0x7f90fc097af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f9100868640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f90fc135590 con 0x7f90fc129160 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f9100868640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f90fc135410 con 0x7f90fc129ab0 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f9100868640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f90fc135710 con 0x7f90fc12c9f0 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f90fa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90fc129160 0x7f90fc097070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f90f9d9b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f90fc129ab0 0x7f90fc0975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f90fad9d640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f90fc12c9f0 0x7f90fc097af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f90fa59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90fc129160 0x7f90fc097070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44340/0 (socket says 172.21.15.27:44340) 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f90f9d9b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f90fc129ab0 0x7f90fc0975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:46352/0 (socket says 172.21.15.27:46352) 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.105+0000 7f90fa59c640 1 -- 172.21.15.27:0/1564924573 learned_addr learned my addr 172.21.15.27:0/1564924573 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:56.109 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f90fa59c640 1 -- 172.21.15.27:0/1564924573 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f90fc12c9f0 msgr2=0x7f90fc097af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f90fa59c640 1 --2- 172.21.15.27:0/1564924573 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f90fc12c9f0 0x7f90fc097af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f90fa59c640 1 -- 172.21.15.27:0/1564924573 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f90fc129ab0 msgr2=0x7f90fc0975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f90fa59c640 1 --2- 172.21.15.27:0/1564924573 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f90fc129ab0 0x7f90fc0975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f90fa59c640 1 -- 172.21.15.27:0/1564924573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90fc131850 con 0x7f90fc129160 2024-09-21T03:28:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f90fad9d640 1 --2- 172.21.15.27:0/1564924573 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f90fc12c9f0 0x7f90fc097af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f90fa59c640 1 --2- 172.21.15.27:0/1564924573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90fc129160 0x7f90fc097070 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00ea20 tx=0x7f90ec00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f90eaffd640 1 -- 172.21.15.27:0/1564924573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec00ce40 con 0x7f90fc129160 2024-09-21T03:28:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f9100868640 1 -- 172.21.15.27:0/1564924573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90fc1320f0 con 0x7f90fc129160 2024-09-21T03:28:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f90eaffd640 1 -- 172.21.15.27:0/1564924573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f90ec004020 con 0x7f90fc129160 2024-09-21T03:28:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f90eaffd640 1 -- 172.21.15.27:0/1564924573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec004470 con 0x7f90fc129160 2024-09-21T03:28:56.110 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.106+0000 7f9100868640 1 -- 172.21.15.27:0/1564924573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90fc132e50 con 0x7f90fc129160 2024-09-21T03:28:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.107+0000 7f9100868640 1 -- 172.21.15.27:0/1564924573 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f90fc129570 con 0x7f90fc129160 2024-09-21T03:28:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.107+0000 7f90eaffd640 1 -- 172.21.15.27:0/1564924573 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f90ec0035f0 con 0x7f90fc129160 2024-09-21T03:28:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.108+0000 7f90eaffd640 1 -- 172.21.15.27:0/1564924573 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f90ec069930 con 0x7f90fc129160 2024-09-21T03:28:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.108+0000 7f90eaffd640 1 -- 172.21.15.27:0/1564924573 --> v1:172.21.15.27:6809/2016211836 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90ec021e50 con 0x7f90dc057db0 2024-09-21T03:28:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.108+0000 7f90eaffd640 1 -- 172.21.15.27:0/1564924573 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f90ec06e2c0 con 0x7f90fc129160 2024-09-21T03:28:56.111 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.109+0000 7f90eaffd640 1 -- 172.21.15.27:0/1564924573 <== osd.2 v1:172.21.15.27:6809/2016211836 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f90ec021e50 con 0x7f90dc057db0 2024-09-21T03:28:56.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.121+0000 7f9100868640 1 -- 172.21.15.27:0/1564924573 --> v1:172.21.15.27:6809/2016211836 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f90fc133360 con 0x7f90dc057db0 2024-09-21T03:28:56.124 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.122+0000 7f90eaffd640 1 -- 172.21.15.27:0/1564924573 <== osd.2 v1:172.21.15.27:6809/2016211836 2 ==== command_reply(tid 2: 0 ) ==== 8+0+565 (unknown 0 0 125856537) 0x7f90fc133360 con 0x7f90dc057db0 2024-09-21T03:28:56.124 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-21T03:28:56.124 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-21T03:28:56.124 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-21T03:28:56.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.122+0000 7f9100868640 1 -- 172.21.15.27:0/1564924573 >> v1:172.21.15.27:6809/2016211836 conn(0x7f90dc057db0 legacy=0x7f90dc05a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.122+0000 7f9100868640 1 -- 172.21.15.27:0/1564924573 >> v1:172.21.15.27:6800/2405211194 conn(0x7f90dc0514e0 legacy=0x7f90dc0539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.122+0000 7f9100868640 1 -- 172.21.15.27:0/1564924573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90fc129160 msgr2=0x7f90fc097070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.125 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.122+0000 7f9100868640 1 --2- 172.21.15.27:0/1564924573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f90fc129160 0x7f90fc097070 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00ea20 tx=0x7f90ec00eee0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.122+0000 7f90fad9d640 1 -- 172.21.15.27:0/1564924573 reap_dead start 2024-09-21T03:28:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.122+0000 7f9100868640 1 -- 172.21.15.27:0/1564924573 shutdown_connections 2024-09-21T03:28:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.122+0000 7f9100868640 1 -- 172.21.15.27:0/1564924573 >> 172.21.15.27:0/1564924573 conn(0x7f90fc09bcb0 msgr2=0x7f90fc099ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.122+0000 7f9100868640 1 -- 172.21.15.27:0/1564924573 shutdown_connections 2024-09-21T03:28:56.126 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.122+0000 7f9100868640 1 -- 172.21.15.27:0/1564924573 wait complete. 2024-09-21T03:28:56.135 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-21T03:28:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.248+0000 7fbadf758640 1 -- 172.21.15.27:0/273349546 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbad812f630 msgr2=0x7fbad812fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.248+0000 7fbadf758640 1 --2- 172.21.15.27:0/273349546 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbad812f630 0x7fbad812fb60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009860 tx=0x7fbad402a460 comp rx=0 tx=0).stop 2024-09-21T03:28:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.249+0000 7fbadf758640 1 -- 172.21.15.27:0/273349546 shutdown_connections 2024-09-21T03:28:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.249+0000 7fbadf758640 1 --2- 172.21.15.27:0/273349546 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbad8070ec0 0x7fbad81300a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.249+0000 7fbadf758640 1 --2- 172.21.15.27:0/273349546 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbad812f630 0x7fbad812fb60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.249+0000 7fbadf758640 1 -- 172.21.15.27:0/273349546 >> 172.21.15.27:0/273349546 conn(0x7fbad80a2050 msgr2=0x7fbad80a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:56.252 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.249+0000 7fbadf758640 1 -- 172.21.15.27:0/273349546 shutdown_connections 2024-09-21T03:28:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.249+0000 7fbadf758640 1 -- 172.21.15.27:0/273349546 wait complete. 2024-09-21T03:28:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.250+0000 7fbadf758640 1 Processor -- start 2024-09-21T03:28:56.253 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.250+0000 7fbadf758640 1 -- start start 2024-09-21T03:28:56.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.250+0000 7fbadf758640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbad8070ec0 0x7fbad81c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.250+0000 7fbadf758640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad812c030 0x7fbad81c5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.250+0000 7fbadf758640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbad812f630 0x7fbad81c6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.250+0000 7fbadf758640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fbad81385c0 con 0x7fbad812c030 2024-09-21T03:28:56.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.250+0000 7fbadf758640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fbad8138440 con 0x7fbad8070ec0 2024-09-21T03:28:56.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.250+0000 7fbadf758640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fbad8138740 con 0x7fbad812f630 2024-09-21T03:28:56.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbadccf3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad812c030 0x7fbad81c5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbadd4f4640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbad8070ec0 0x7fbad81c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbaddcf5640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbad812f630 0x7fbad81c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.257 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbadd4f4640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbad8070ec0 0x7fbad81c5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:46370/0 (socket says 172.21.15.27:46370) 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbadccf3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad812c030 0x7fbad81c5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44346/0 (socket says 172.21.15.27:44346) 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbadd4f4640 1 -- 172.21.15.27:0/2574234930 learned_addr learned my addr 172.21.15.27:0/2574234930 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbaddcf5640 1 -- 172.21.15.27:0/2574234930 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbad8070ec0 msgr2=0x7fbad81c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbaddcf5640 1 --2- 172.21.15.27:0/2574234930 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbad8070ec0 0x7fbad81c5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbaddcf5640 1 -- 172.21.15.27:0/2574234930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad812c030 msgr2=0x7fbad81c5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbaddcf5640 1 --2- 172.21.15.27:0/2574234930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad812c030 0x7fbad81c5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbaddcf5640 1 -- 172.21.15.27:0/2574234930 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad809a650 con 0x7fbad812f630 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbadd4f4640 1 --2- 172.21.15.27:0/2574234930 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbad8070ec0 0x7fbad81c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbaddcf5640 1 --2- 172.21.15.27:0/2574234930 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbad812f630 0x7fbad81c6490 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbad000ca70 tx=0x7fbad000cf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbacdffb640 1 -- 172.21.15.27:0/2574234930 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad0016020 con 0x7fbad812f630 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbacdffb640 1 -- 172.21.15.27:0/2574234930 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fbad000ac80 con 0x7fbad812f630 2024-09-21T03:28:56.258 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbadf758640 1 -- 172.21.15.27:0/2574234930 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad809b470 con 0x7fbad812f630 2024-09-21T03:28:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbacdffb640 1 -- 172.21.15.27:0/2574234930 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad0012680 con 0x7fbad812f630 2024-09-21T03:28:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.251+0000 7fbadf758640 1 -- 172.21.15.27:0/2574234930 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad8098860 con 0x7fbad812f630 2024-09-21T03:28:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.253+0000 7fbadf758640 1 -- 172.21.15.27:0/2574234930 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad8074450 con 0x7fbad812f630 2024-09-21T03:28:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.253+0000 7fbacdffb640 1 -- 172.21.15.27:0/2574234930 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7fbad0019020 con 0x7fbad812f630 2024-09-21T03:28:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.253+0000 7fbacdffb640 1 -- 172.21.15.27:0/2574234930 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7fbad0069570 con 0x7fbad812f630 2024-09-21T03:28:56.259 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.257+0000 7fbacdffb640 1 -- 172.21.15.27:0/2574234930 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fbad8074450 con 0x7fbad812f630 2024-09-21T03:28:56.420 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.418+0000 7fbadf758640 1 -- 172.21.15.27:0/2574234930 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fbad812c440 con 0x7fbabc0514e0 2024-09-21T03:28:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.418+0000 7fbacdffb640 1 -- 172.21.15.27:0/2574234930 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7fbad812c440 con 0x7fbabc0514e0 2024-09-21T03:28:56.421 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 2 to deep-scrub 2024-09-21T03:28:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.420+0000 7fbadf758640 1 -- 172.21.15.27:0/2574234930 >> v1:172.21.15.27:6800/2405211194 conn(0x7fbabc0514e0 legacy=0x7fbabc0539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.420+0000 7fbadf758640 1 -- 172.21.15.27:0/2574234930 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbad812f630 msgr2=0x7fbad81c6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.420+0000 7fbadf758640 1 --2- 172.21.15.27:0/2574234930 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbad812f630 0x7fbad81c6490 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbad000ca70 tx=0x7fbad000cf30 comp rx=0 tx=0).stop 2024-09-21T03:28:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.420+0000 7fbadf758640 1 -- 172.21.15.27:0/2574234930 shutdown_connections 2024-09-21T03:28:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.420+0000 7fbadf758640 1 --2- 172.21.15.27:0/2574234930 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fbad812f630 0x7fbad81c6490 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.423 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.420+0000 7fbadf758640 1 --2- 172.21.15.27:0/2574234930 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbad812c030 0x7fbad81c5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.420+0000 7fbadf758640 1 --2- 172.21.15.27:0/2574234930 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fbad8070ec0 0x7fbad81c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.420+0000 7fbadf758640 1 -- 172.21.15.27:0/2574234930 >> 172.21.15.27:0/2574234930 conn(0x7fbad80a2050 msgr2=0x7fbad8097520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:56.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.421+0000 7fbadf758640 1 -- 172.21.15.27:0/2574234930 shutdown_connections 2024-09-21T03:28:56.424 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.421+0000 7fbadf758640 1 -- 172.21.15.27:0/2574234930 wait complete. 2024-09-21T03:28:56.438 INFO:tasks.ceph:Scrubbing osd.3 2024-09-21T03:28:56.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-21T03:28:56.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.554+0000 7f961520a640 1 -- 172.21.15.27:0/1088633113 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9610071940 msgr2=0x7f961012c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.554+0000 7f961520a640 1 --2- 172.21.15.27:0/1088633113 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9610071940 0x7f961012c4b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9600009860 tx=0x7f960002a3d0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.557 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.554+0000 7f961520a640 1 -- 172.21.15.27:0/1088633113 shutdown_connections 2024-09-21T03:28:56.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.554+0000 7f961520a640 1 --2- 172.21.15.27:0/1088633113 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f961012ce00 0x7f961012fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.554+0000 7f961520a640 1 --2- 172.21.15.27:0/1088633113 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9610071940 0x7f961012c4b0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.554+0000 7f961520a640 1 -- 172.21.15.27:0/1088633113 >> 172.21.15.27:0/1088633113 conn(0x7f96100a1f10 msgr2=0x7f96100a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:56.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.554+0000 7f961520a640 1 -- 172.21.15.27:0/1088633113 shutdown_connections 2024-09-21T03:28:56.558 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.554+0000 7f961520a640 1 -- 172.21.15.27:0/1088633113 wait complete. 2024-09-21T03:28:56.561 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f961520a640 1 Processor -- start 2024-09-21T03:28:56.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f961520a640 1 -- start start 2024-09-21T03:28:56.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f961520a640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9610071940 0x7f961009b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f961520a640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f961012c9f0 0x7f961009b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f961520a640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961012ce00 0x7f961009bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f961520a640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9610138710 con 0x7f961012ce00 2024-09-21T03:28:56.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f961520a640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f9610138590 con 0x7f9610071940 2024-09-21T03:28:56.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f961520a640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f9610138890 con 0x7f961012c9f0 2024-09-21T03:28:56.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f960e59c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f961012c9f0 0x7f961009b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f960ed9d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9610071940 0x7f961009b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.562 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f960f59e640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961012ce00 0x7f961009bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f960ed9d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9610071940 0x7f961009b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:46384/0 (socket says 172.21.15.27:46384) 2024-09-21T03:28:56.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f960e59c640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f961012c9f0 0x7f961009b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:35712/0 (socket says 172.21.15.27:35712) 2024-09-21T03:28:56.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.556+0000 7f960ed9d640 1 -- 172.21.15.27:0/1645975753 learned_addr learned my addr 172.21.15.27:0/1645975753 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:56.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f960f59e640 1 -- 172.21.15.27:0/1645975753 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f961012c9f0 msgr2=0x7f961009b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f960f59e640 1 --2- 172.21.15.27:0/1645975753 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f961012c9f0 0x7f961009b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f960f59e640 1 -- 172.21.15.27:0/1645975753 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9610071940 msgr2=0x7f961009b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f960f59e640 1 --2- 172.21.15.27:0/1645975753 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9610071940 0x7f961009b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f960f59e640 1 -- 172.21.15.27:0/1645975753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9610133a10 con 0x7f961012ce00 2024-09-21T03:28:56.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f960e59c640 1 --2- 172.21.15.27:0/1645975753 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f961012c9f0 0x7f961009b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:56.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f960ed9d640 1 --2- 172.21.15.27:0/1645975753 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9610071940 0x7f961009b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:56.563 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f960f59e640 1 --2- 172.21.15.27:0/1645975753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961012ce00 0x7f961009bf00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f960800eb50 tx=0x7f960800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f95f77fe640 1 -- 172.21.15.27:0/1645975753 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9608003b70 con 0x7f961012ce00 2024-09-21T03:28:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f95f77fe640 1 -- 172.21.15.27:0/1645975753 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9608003d10 con 0x7f961012ce00 2024-09-21T03:28:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f961520a640 1 -- 172.21.15.27:0/1645975753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9610134860 con 0x7f961012ce00 2024-09-21T03:28:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f95f77fe640 1 -- 172.21.15.27:0/1645975753 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9608004280 con 0x7f961012ce00 2024-09-21T03:28:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.557+0000 7f961520a640 1 -- 172.21.15.27:0/1645975753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9610131b80 con 0x7f961012ce00 2024-09-21T03:28:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.558+0000 7f961520a640 1 -- 172.21.15.27:0/1645975753 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f961012c4b0 con 0x7f961012ce00 2024-09-21T03:28:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.559+0000 7f95f77fe640 1 -- 172.21.15.27:0/1645975753 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f9608004420 con 0x7f961012ce00 2024-09-21T03:28:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.559+0000 7f95f77fe640 1 -- 172.21.15.27:0/1645975753 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f9608069be0 con 0x7f961012ce00 2024-09-21T03:28:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.559+0000 7f95f77fe640 1 -- 172.21.15.27:0/1645975753 --> v1:172.21.15.27:6813/1188450781 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9608005a20 con 0x7f95fc057df0 2024-09-21T03:28:56.564 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.559+0000 7f95f77fe640 1 -- 172.21.15.27:0/1645975753 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f960806c2e0 con 0x7f961012ce00 2024-09-21T03:28:56.565 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.560+0000 7f95f77fe640 1 -- 172.21.15.27:0/1645975753 <== osd.3 v1:172.21.15.27:6813/1188450781 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f9608005a20 con 0x7f95fc057df0 2024-09-21T03:28:56.575 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.573+0000 7f961520a640 1 -- 172.21.15.27:0/1645975753 --> v1:172.21.15.27:6813/1188450781 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9610129050 con 0x7f95fc057df0 2024-09-21T03:28:56.576 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.574+0000 7f95f77fe640 1 -- 172.21.15.27:0/1645975753 <== osd.3 v1:172.21.15.27:6813/1188450781 2 ==== command_reply(tid 2: 0 ) ==== 8+0+565 (unknown 0 0 125856537) 0x7f9610129050 con 0x7f95fc057df0 2024-09-21T03:28:56.576 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2024-09-21T03:28:56.576 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-21T03:28:56.577 INFO:teuthology.orchestra.run.smithi027.stdout:} 2024-09-21T03:28:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.574+0000 7f961520a640 1 -- 172.21.15.27:0/1645975753 >> v1:172.21.15.27:6813/1188450781 conn(0x7f95fc057df0 legacy=0x7f95fc05a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.574+0000 7f961520a640 1 -- 172.21.15.27:0/1645975753 >> v1:172.21.15.27:6800/2405211194 conn(0x7f95fc0514b0 legacy=0x7f95fc0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.574+0000 7f961520a640 1 -- 172.21.15.27:0/1645975753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961012ce00 msgr2=0x7f961009bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.574+0000 7f961520a640 1 --2- 172.21.15.27:0/1645975753 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f961012ce00 0x7f961009bf00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f960800eb50 tx=0x7f960800c6a0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.574+0000 7f960f59e640 1 -- 172.21.15.27:0/1645975753 reap_dead start 2024-09-21T03:28:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.574+0000 7f961520a640 1 -- 172.21.15.27:0/1645975753 shutdown_connections 2024-09-21T03:28:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.574+0000 7f961520a640 1 -- 172.21.15.27:0/1645975753 >> 172.21.15.27:0/1645975753 conn(0x7f96100a1f10 msgr2=0x7f961012ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:56.578 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.574+0000 7f961520a640 1 -- 172.21.15.27:0/1645975753 shutdown_connections 2024-09-21T03:28:56.579 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.574+0000 7f961520a640 1 -- 172.21.15.27:0/1645975753 wait complete. 2024-09-21T03:28:56.587 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-21T03:28:56.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.700+0000 7fc996956640 1 -- 172.21.15.27:0/1255724267 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc990132ea0 msgr2=0x7fc9901352c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.700+0000 7fc996956640 1 --2- 172.21.15.27:0/1255724267 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc990132ea0 0x7fc9901352c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc988005fd0 tx=0x7fc98802a820 comp rx=0 tx=0).stop 2024-09-21T03:28:56.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.700+0000 7fc996956640 1 -- 172.21.15.27:0/1255724267 shutdown_connections 2024-09-21T03:28:56.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.700+0000 7fc996956640 1 --2- 172.21.15.27:0/1255724267 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc990132ea0 0x7fc9901352c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.700+0000 7fc996956640 1 --2- 172.21.15.27:0/1255724267 >> v2:172.21.15.27:3300/0 conn(0x7fc99012aaa0 0x7fc990132960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.703 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.700+0000 7fc996956640 1 -- 172.21.15.27:0/1255724267 >> 172.21.15.27:0/1255724267 conn(0x7fc99009fa20 msgr2=0x7fc99009fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:56.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.700+0000 7fc996956640 1 -- 172.21.15.27:0/1255724267 shutdown_connections 2024-09-21T03:28:56.704 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.700+0000 7fc996956640 1 -- 172.21.15.27:0/1255724267 wait complete. 2024-09-21T03:28:56.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.701+0000 7fc996956640 1 Processor -- start 2024-09-21T03:28:56.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc996956640 1 -- start start 2024-09-21T03:28:56.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc996956640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc990073c10 0x7fc990099460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc996956640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc99012aaa0 0x7fc9900999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc996956640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc990132ea0 0x7fc99006faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:56.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc996956640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7fc990136550 con 0x7fc990132ea0 2024-09-21T03:28:56.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc996956640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7fc9901363d0 con 0x7fc99012aaa0 2024-09-21T03:28:56.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc996956640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7fc9901366d0 con 0x7fc990073c10 2024-09-21T03:28:56.706 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc98f7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc99012aaa0 0x7fc9900999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc98ffff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc990073c10 0x7fc990099460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc994ef3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc990132ea0 0x7fc99006faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc98f7fe640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc99012aaa0 0x7fc9900999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:46394/0 (socket says 172.21.15.27:46394) 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc994ef3640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc990132ea0 0x7fc99006faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44374/0 (socket says 172.21.15.27:44374) 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc98f7fe640 1 -- 172.21.15.27:0/44188982 learned_addr learned my addr 172.21.15.27:0/44188982 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.702+0000 7fc98ffff640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc990073c10 0x7fc990099460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:35728/0 (socket says 172.21.15.27:35728) 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc98f7fe640 1 -- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc990073c10 msgr2=0x7fc990099460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc98f7fe640 1 --2- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc990073c10 0x7fc990099460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc98f7fe640 1 -- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc990132ea0 msgr2=0x7fc99006faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc98f7fe640 1 --2- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc990132ea0 0x7fc99006faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc98f7fe640 1 -- 172.21.15.27:0/44188982 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc990127d30 con 0x7fc99012aaa0 2024-09-21T03:28:56.707 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc994ef3640 1 --2- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc990132ea0 0x7fc99006faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc98ffff640 1 --2- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc990073c10 0x7fc990099460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc98f7fe640 1 --2- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc99012aaa0 0x7fc9900999a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc98400e980 tx=0x7fc98400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc98cff9640 1 -- 172.21.15.27:0/44188982 <== mon.1 v2:172.21.15.27:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98400cd20 con 0x7fc99012aaa0 2024-09-21T03:28:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc98cff9640 1 -- 172.21.15.27:0/44188982 <== mon.1 v2:172.21.15.27:3301/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7fc984010430 con 0x7fc99012aaa0 2024-09-21T03:28:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc996956640 1 -- 172.21.15.27:0/44188982 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc990099ee0 con 0x7fc99012aaa0 2024-09-21T03:28:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc98cff9640 1 -- 172.21.15.27:0/44188982 <== mon.1 v2:172.21.15.27:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984010860 con 0x7fc99012aaa0 2024-09-21T03:28:56.708 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.703+0000 7fc996956640 1 -- 172.21.15.27:0/44188982 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc990070030 con 0x7fc99012aaa0 2024-09-21T03:28:56.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.704+0000 7fc996956640 1 -- 172.21.15.27:0/44188982 --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc99012a390 con 0x7fc99012aaa0 2024-09-21T03:28:56.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.704+0000 7fc98cff9640 1 -- 172.21.15.27:0/44188982 <== mon.1 v2:172.21.15.27:3301/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7fc98401f020 con 0x7fc99012aaa0 2024-09-21T03:28:56.709 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.705+0000 7fc98cff9640 1 -- 172.21.15.27:0/44188982 <== mon.1 v2:172.21.15.27:3301/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7fc984069730 con 0x7fc99012aaa0 2024-09-21T03:28:56.711 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.709+0000 7fc98cff9640 1 -- 172.21.15.27:0/44188982 <== mon.1 v2:172.21.15.27:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7fc9840687e0 con 0x7fc99012aaa0 2024-09-21T03:28:56.871 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.869+0000 7fc996956640 1 -- 172.21.15.27:0/44188982 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fc99012f690 con 0x7fc97c0514b0 2024-09-21T03:28:56.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.869+0000 7fc98cff9640 1 -- 172.21.15.27:0/44188982 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7fc980001570 con 0x7fc97c0514b0 2024-09-21T03:28:56.875 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 3 to deep-scrub 2024-09-21T03:28:56.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.871+0000 7fc996956640 1 -- 172.21.15.27:0/44188982 >> v1:172.21.15.27:6800/2405211194 conn(0x7fc97c0514b0 legacy=0x7fc97c0539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.871+0000 7fc996956640 1 -- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc99012aaa0 msgr2=0x7fc9900999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:56.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.871+0000 7fc996956640 1 --2- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc99012aaa0 0x7fc9900999a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc98400e980 tx=0x7fc98400ee40 comp rx=0 tx=0).stop 2024-09-21T03:28:56.875 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.871+0000 7fc996956640 1 -- 172.21.15.27:0/44188982 shutdown_connections 2024-09-21T03:28:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.871+0000 7fc996956640 1 --2- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc990132ea0 0x7fc99006faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.871+0000 7fc996956640 1 --2- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7fc99012aaa0 0x7fc9900999a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.871+0000 7fc996956640 1 --2- 172.21.15.27:0/44188982 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7fc990073c10 0x7fc990099460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.871+0000 7fc996956640 1 -- 172.21.15.27:0/44188982 >> 172.21.15.27:0/44188982 conn(0x7fc99009fa20 msgr2=0x7fc990127130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.871+0000 7fc996956640 1 -- 172.21.15.27:0/44188982 shutdown_connections 2024-09-21T03:28:56.876 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.871+0000 7fc996956640 1 -- 172.21.15.27:0/44188982 wait complete. 2024-09-21T03:28:56.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T03:28:56.999 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.997+0000 7f9d310e8640 1 -- 172.21.15.27:0/3609431625 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9d2c130280 msgr2=0x7f9d2c12c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.997+0000 7f9d310e8640 1 --2- 172.21.15.27:0/3609431625 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9d2c130280 0x7f9d2c12c650 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9d24005fd0 tx=0x7f9d2402a6e0 comp rx=0 tx=0).stop 2024-09-21T03:28:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.997+0000 7f9d310e8640 1 -- 172.21.15.27:0/3609431625 shutdown_connections 2024-09-21T03:28:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.997+0000 7f9d310e8640 1 --2- 172.21.15.27:0/3609431625 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9d2c130280 0x7f9d2c12c650 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.997+0000 7f9d310e8640 1 --2- 172.21.15.27:0/3609431625 >> v2:172.21.15.27:3300/0 conn(0x7f9d2c12f900 0x7f9d2c12c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.997+0000 7f9d310e8640 1 --2- 172.21.15.27:0/3609431625 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9d2c071a30 0x7f9d2c070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.997+0000 7f9d310e8640 1 -- 172.21.15.27:0/3609431625 >> 172.21.15.27:0/3609431625 conn(0x7f9d2c0a2050 msgr2=0x7f9d2c0a2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.997+0000 7f9d310e8640 1 -- 172.21.15.27:0/3609431625 shutdown_connections 2024-09-21T03:28:57.000 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.997+0000 7f9d310e8640 1 -- 172.21.15.27:0/3609431625 wait complete. 2024-09-21T03:28:57.002 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.998+0000 7f9d310e8640 1 Processor -- start 2024-09-21T03:28:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.998+0000 7f9d310e8640 1 -- start start 2024-09-21T03:28:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d310e8640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9d2c071a30 0x7f9d2c1c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d310e8640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c12f900 0x7f9d2c1c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d310e8640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9d2c130280 0x7f9d2c1c6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:28:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d310e8640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2c137440 con 0x7f9d2c12f900 2024-09-21T03:28:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d310e8640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f9d2c1372c0 con 0x7f9d2c071a30 2024-09-21T03:28:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d310e8640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f9d2c1375c0 con 0x7f9d2c130280 2024-09-21T03:28:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d2a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c12f900 0x7f9d2c1c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d2ad9d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9d2c071a30 0x7f9d2c1c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d2b59e640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9d2c130280 0x7f9d2c1c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:28:57.003 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d2a59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c12f900 0x7f9d2c1c5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:44390/0 (socket says 172.21.15.27:44390) 2024-09-21T03:28:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d2b59e640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9d2c130280 0x7f9d2c1c6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:35738/0 (socket says 172.21.15.27:35738) 2024-09-21T03:28:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d2ad9d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9d2c071a30 0x7f9d2c1c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:46408/0 (socket says 172.21.15.27:46408) 2024-09-21T03:28:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d2a59c640 1 -- 172.21.15.27:0/3992992859 learned_addr learned my addr 172.21.15.27:0/3992992859 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:28:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:56.999+0000 7f9d2b59e640 1 -- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9d2c071a30 msgr2=0x7f9d2c1c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d2b59e640 1 --2- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9d2c071a30 0x7f9d2c1c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d2b59e640 1 -- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c12f900 msgr2=0x7f9d2c1c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d2b59e640 1 --2- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c12f900 0x7f9d2c1c5d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d2b59e640 1 -- 172.21.15.27:0/3992992859 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d2c134050 con 0x7f9d2c130280 2024-09-21T03:28:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d2a59c640 1 --2- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c12f900 0x7f9d2c1c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:57.004 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d2ad9d640 1 --2- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9d2c071a30 0x7f9d2c1c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:28:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d2b59e640 1 --2- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9d2c130280 0x7f9d2c1c6290 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009000 tx=0x7f9d24032de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:28:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d137fe640 1 -- 172.21.15.27:0/3992992859 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d24041020 con 0x7f9d2c130280 2024-09-21T03:28:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d137fe640 1 -- 172.21.15.27:0/3992992859 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f9d24004040 con 0x7f9d2c130280 2024-09-21T03:28:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d310e8640 1 -- 172.21.15.27:0/3992992859 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d2c134bd0 con 0x7f9d2c130280 2024-09-21T03:28:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d137fe640 1 -- 172.21.15.27:0/3992992859 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d24036040 con 0x7f9d2c130280 2024-09-21T03:28:57.005 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.000+0000 7f9d310e8640 1 -- 172.21.15.27:0/3992992859 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d2c1350e0 con 0x7f9d2c130280 2024-09-21T03:28:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.001+0000 7f9d310e8640 1 -- 172.21.15.27:0/3992992859 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d2c070cb0 con 0x7f9d2c130280 2024-09-21T03:28:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.001+0000 7f9d137fe640 1 -- 172.21.15.27:0/3992992859 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f9d24044020 con 0x7f9d2c130280 2024-09-21T03:28:57.006 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.002+0000 7f9d137fe640 1 -- 172.21.15.27:0/3992992859 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f9d24089e80 con 0x7f9d2c130280 2024-09-21T03:28:57.007 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.005+0000 7f9d137fe640 1 -- 172.21.15.27:0/3992992859 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f9d2404b030 con 0x7f9d2c130280 2024-09-21T03:28:57.165 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.163+0000 7f9d310e8640 1 -- 172.21.15.27:0/3992992859 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9d2c1345e0 con 0x7f9d180514b0 2024-09-21T03:28:57.166 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.164+0000 7f9d137fe640 1 -- 172.21.15.27:0/3992992859 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31980 (unknown 2965378022 0 99470882) 0x7f9d2c1345e0 con 0x7f9d180514b0 2024-09-21T03:28:57.167 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:28:57.167 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-21T03:28:57.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.165+0000 7f9d310e8640 1 -- 172.21.15.27:0/3992992859 >> v1:172.21.15.27:6800/2405211194 conn(0x7f9d180514b0 legacy=0x7f9d180539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:57.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.165+0000 7f9d310e8640 1 -- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9d2c130280 msgr2=0x7f9d2c1c6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:28:57.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.165+0000 7f9d310e8640 1 --2- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9d2c130280 0x7f9d2c1c6290 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9d24009000 tx=0x7f9d24032de0 comp rx=0 tx=0).stop 2024-09-21T03:28:57.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.166+0000 7f9d310e8640 1 -- 172.21.15.27:0/3992992859 shutdown_connections 2024-09-21T03:28:57.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.166+0000 7f9d310e8640 1 --2- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f9d2c130280 0x7f9d2c1c6290 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:57.168 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.166+0000 7f9d310e8640 1 --2- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9d2c12f900 0x7f9d2c1c5d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.166+0000 7f9d310e8640 1 --2- 172.21.15.27:0/3992992859 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f9d2c071a30 0x7f9d2c1c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:28:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.166+0000 7f9d310e8640 1 -- 172.21.15.27:0/3992992859 >> 172.21.15.27:0/3992992859 conn(0x7f9d2c0a2050 msgr2=0x7f9d2c12af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:28:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.166+0000 7f9d310e8640 1 -- 172.21.15.27:0/3992992859 shutdown_connections 2024-09-21T03:28:57.169 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:28:57.166+0000 7f9d310e8640 1 -- 172.21.15.27:0/3992992859 wait complete. 2024-09-21T03:28:57.179 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-21T03:28:56.300818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":151,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107,"ondisk_log_size":107,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":116608,"kb_used_data":2280,"kb_used_omap":17,"kb_used_meta":114222,"kb_avail":377370752,"statfs":{"total":386547056640,"available":386427650048,"internally_reserved":0,"allocated":2334720,"data_stored":1576729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17729,"internal_metadata":116964031},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.319949"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982313+0000","last_change":"2024-09-21T03:28:44.676220+0000","last_active":"2024-09-21T03:28:47.982313+0000","last_peered":"2024-09-21T03:28:47.982313+0000","last_clean":"2024-09-21T03:28:47.982313+0000","last_became_active":"2024-09-21T03:28:44.675664+0000","last_became_peered":"2024-09-21T03:28:44.675664+0000","last_unstale":"2024-09-21T03:28:47.982313+0000","last_undegraded":"2024-09-21T03:28:47.982313+0000","last_fullsized":"2024-09-21T03:28:47.982313+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:02.081362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393285,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249352+0000","last_change":"2024-09-21T03:28:40.835755+0000","last_active":"2024-09-21T03:28:49.249352+0000","last_peered":"2024-09-21T03:28:49.249352+0000","last_clean":"2024-09-21T03:28:49.249352+0000","last_became_active":"2024-09-21T03:28:40.835171+0000","last_became_peered":"2024-09-21T03:28:40.835171+0000","last_unstale":"2024-09-21T03:28:49.249352+0000","last_undegraded":"2024-09-21T03:28:49.249352+0000","last_fullsized":"2024-09-21T03:28:49.249352+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:17:17.586168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044937200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":49,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982197+0000","last_change":"2024-09-21T03:28:44.676192+0000","last_active":"2024-09-21T03:28:47.982197+0000","last_peered":"2024-09-21T03:28:47.982197+0000","last_clean":"2024-09-21T03:28:47.982197+0000","last_became_active":"2024-09-21T03:28:44.675680+0000","last_became_peered":"2024-09-21T03:28:44.675680+0000","last_unstale":"2024-09-21T03:28:47.982197+0000","last_undegraded":"2024-09-21T03:28:47.982197+0000","last_fullsized":"2024-09-21T03:28:47.982197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:21:10.791750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249310+0000","last_change":"2024-09-21T03:28:40.835586+0000","last_active":"2024-09-21T03:28:49.249310+0000","last_peered":"2024-09-21T03:28:49.249310+0000","last_clean":"2024-09-21T03:28:49.249310+0000","last_became_active":"2024-09-21T03:28:40.835243+0000","last_became_peered":"2024-09-21T03:28:40.835243+0000","last_unstale":"2024-09-21T03:28:49.249310+0000","last_undegraded":"2024-09-21T03:28:49.249310+0000","last_fullsized":"2024-09-21T03:28:49.249310+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:31.997034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049584300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":47,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982077+0000","last_change":"2024-09-21T03:28:43.852388+0000","last_active":"2024-09-21T03:28:47.982077+0000","last_peered":"2024-09-21T03:28:47.982077+0000","last_clean":"2024-09-21T03:28:47.982077+0000","last_became_active":"2024-09-21T03:28:43.851555+0000","last_became_peered":"2024-09-21T03:28:43.851555+0000","last_unstale":"2024-09-21T03:28:47.982077+0000","last_undegraded":"2024-09-21T03:28:47.982077+0000","last_fullsized":"2024-09-21T03:28:47.982077+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T03:37:19.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010094349999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":45,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:48.996126+0000","last_change":"2024-09-21T03:28:43.850674+0000","last_active":"2024-09-21T03:28:48.996126+0000","last_peered":"2024-09-21T03:28:48.996126+0000","last_clean":"2024-09-21T03:28:48.996126+0000","last_became_active":"2024-09-21T03:28:43.850058+0000","last_became_peered":"2024-09-21T03:28:43.850058+0000","last_unstale":"2024-09-21T03:28:48.996126+0000","last_undegraded":"2024-09-21T03:28:48.996126+0000","last_fullsized":"2024-09-21T03:28:48.996126+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:58:47.763461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056217599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":45,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.853213+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.851256+0000","last_became_peered":"2024-09-21T03:28:43.851256+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:04:03.534378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045280299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":46,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-21T03:28:47.982067+0000","last_change":"2024-09-21T03:28:43.850214+0000","last_active":"2024-09-21T03:28:47.982067+0000","last_peered":"2024-09-21T03:28:47.982067+0000","last_clean":"2024-09-21T03:28:47.982067+0000","last_became_active":"2024-09-21T03:28:43.849169+0000","last_became_peered":"2024-09-21T03:28:43.849169+0000","last_unstale":"2024-09-21T03:28:47.982067+0000","last_undegraded":"2024-09-21T03:28:47.982067+0000","last_fullsized":"2024-09-21T03:28:47.982067+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:12.253174+0000","last_clean_scrub_stamp":"2024-09-21T03:27:12.253174+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:10:34.860226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"21'104","reported_seq":201,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:49.249233+0000","last_change":"2024-09-21T03:28:40.835338+0000","last_active":"2024-09-21T03:28:49.249233+0000","last_peered":"2024-09-21T03:28:49.249233+0000","last_clean":"2024-09-21T03:28:49.249233+0000","last_became_active":"2024-09-21T03:28:40.834765+0000","last_became_peered":"2024-09-21T03:28:40.834765+0000","last_unstale":"2024-09-21T03:28:49.249233+0000","last_undegraded":"2024-09-21T03:28:49.249233+0000","last_fullsized":"2024-09-21T03:28:49.249233+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:27:08.233475+0000","last_clean_scrub_stamp":"2024-09-21T03:27:08.233475+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:31:57.980901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":31,"seq":133143986180,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":276,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":282624,"data_stored":96334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":28176047},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":29,"seq":124554051589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":280,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":286720,"data_stored":99880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":864,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":884736,"data_stored":690267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32435888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964116997,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28404,"kb_used_data":860,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343436,"statfs":{"total":96636764160,"available":96607678464,"internally_reserved":0,"allocated":880640,"data_stored":690248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T03:28:57.183 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-21T03:27:12.253174+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=27, tm_sec=12, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=28, tm_sec=54, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T03:28:57.183 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-21T03:27:12.253174+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=27, tm_sec=12, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=28, tm_sec=54, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T03:28:57.183 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-21T03:27:12.253174+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=27, tm_sec=12, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=28, tm_sec=54, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T03:28:57.183 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-21T03:27:12.253174+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=27, tm_sec=12, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=28, tm_sec=54, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T03:28:57.184 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-21T03:27:12.253174+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=27, tm_sec=12, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=28, tm_sec=54, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T03:28:57.184 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-21T03:27:12.253174+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=27, tm_sec=12, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=28, tm_sec=54, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T03:28:57.184 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-21T03:27:12.253174+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=27, tm_sec=12, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=28, tm_sec=54, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T03:28:57.184 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-21T03:27:12.253174+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=27, tm_sec=12, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=28, tm_sec=54, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T03:28:57.185 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-21T03:27:08.233475+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=27, tm_sec=8, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=3, tm_min=28, tm_sec=54, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T03:28:57.185 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-21T03:29:17.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T03:29:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.315+0000 7f321484b640 1 -- 172.21.15.27:0/1508057314 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3210134eb0 msgr2=0x7f32101372d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:29:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.315+0000 7f321484b640 1 --2- 172.21.15.27:0/1508057314 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3210134eb0 0x7f32101372d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3208009c10 tx=0x7f320802a8a0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.315+0000 7f321484b640 1 -- 172.21.15.27:0/1508057314 shutdown_connections 2024-09-21T03:29:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.315+0000 7f321484b640 1 --2- 172.21.15.27:0/1508057314 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3210134eb0 0x7f32101372d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.316 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.315+0000 7f321484b640 1 --2- 172.21.15.27:0/1508057314 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f321012cbd0 0x7f3210134970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.315+0000 7f321484b640 1 -- 172.21.15.27:0/1508057314 >> 172.21.15.27:0/1508057314 conn(0x7f32100a1d70 msgr2=0x7f32100a2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:29:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.315+0000 7f321484b640 1 -- 172.21.15.27:0/1508057314 shutdown_connections 2024-09-21T03:29:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.315+0000 7f321484b640 1 -- 172.21.15.27:0/1508057314 wait complete. 2024-09-21T03:29:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.316+0000 7f321484b640 1 Processor -- start 2024-09-21T03:29:17.317 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.316+0000 7f321484b640 1 -- start start 2024-09-21T03:29:17.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f321484b640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32100718e0 0x7f321009da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:29:17.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f321484b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f321012cbd0 0x7f321009dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:29:17.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f321484b640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3210134eb0 0x7f321009e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:29:17.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f321484b640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f32101386f0 con 0x7f32100718e0 2024-09-21T03:29:17.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f321484b640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f3210138570 con 0x7f3210134eb0 2024-09-21T03:29:17.321 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f321484b640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f3210138870 con 0x7f321012cbd0 2024-09-21T03:29:17.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f320e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32100718e0 0x7f321009da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:29:17.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f3205d9b640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f321012cbd0 0x7f321009dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:29:17.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f320ed9d640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3210134eb0 0x7f321009e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:29:17.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f320e59c640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32100718e0 0x7f321009da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52592/0 (socket says 172.21.15.27:52592) 2024-09-21T03:29:17.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f320e59c640 1 -- 172.21.15.27:0/262714033 learned_addr learned my addr 172.21.15.27:0/262714033 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:29:17.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f320e59c640 1 -- 172.21.15.27:0/262714033 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f321012cbd0 msgr2=0x7f321009dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:29:17.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f320e59c640 1 --2- 172.21.15.27:0/262714033 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f321012cbd0 0x7f321009dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.322 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f320e59c640 1 -- 172.21.15.27:0/262714033 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3210134eb0 msgr2=0x7f321009e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:29:17.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f320e59c640 1 --2- 172.21.15.27:0/262714033 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3210134eb0 0x7f321009e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f320e59c640 1 -- 172.21.15.27:0/262714033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3210131480 con 0x7f32100718e0 2024-09-21T03:29:17.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.317+0000 7f320ed9d640 1 --2- 172.21.15.27:0/262714033 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3210134eb0 0x7f321009e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:29:17.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.318+0000 7f320e59c640 1 --2- 172.21.15.27:0/262714033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32100718e0 0x7f321009da70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f320000c460 tx=0x7f320000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:29:17.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.318+0000 7f32077fe640 1 -- 172.21.15.27:0/262714033 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3200016020 con 0x7f32100718e0 2024-09-21T03:29:17.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.318+0000 7f32077fe640 1 -- 172.21.15.27:0/262714033 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f3200012940 con 0x7f32100718e0 2024-09-21T03:29:17.323 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.318+0000 7f321484b640 1 -- 172.21.15.27:0/262714033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3210130c00 con 0x7f32100718e0 2024-09-21T03:29:17.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.318+0000 7f32077fe640 1 -- 172.21.15.27:0/262714033 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3200012d90 con 0x7f32100718e0 2024-09-21T03:29:17.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.318+0000 7f321484b640 1 -- 172.21.15.27:0/262714033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32101cbd50 con 0x7f32100718e0 2024-09-21T03:29:17.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.319+0000 7f321484b640 1 -- 172.21.15.27:0/262714033 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f321012c4c0 con 0x7f32100718e0 2024-09-21T03:29:17.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.319+0000 7f32077fe640 1 -- 172.21.15.27:0/262714033 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f3200019020 con 0x7f32100718e0 2024-09-21T03:29:17.324 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.320+0000 7f32077fe640 1 -- 172.21.15.27:0/262714033 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f3200068870 con 0x7f32100718e0 2024-09-21T03:29:17.325 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.323+0000 7f32077fe640 1 -- 172.21.15.27:0/262714033 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f3200020070 con 0x7f32100718e0 2024-09-21T03:29:17.481 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.480+0000 7f321484b640 1 -- 172.21.15.27:0/262714033 --> v1:172.21.15.27:6800/2405211194 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f321006dda0 con 0x7f31f00514b0 2024-09-21T03:29:17.482 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.481+0000 7f32077fe640 1 -- 172.21.15.27:0/262714033 <== mgr.14101 v1:172.21.15.27:6800/2405211194 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31991 (unknown 2965378022 0 4290759537) 0x7f321006dda0 con 0x7f31f00514b0 2024-09-21T03:29:17.482 INFO:teuthology.orchestra.run.smithi027.stdout: 2024-09-21T03:29:17.483 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2024-09-21T03:29:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.483+0000 7f321484b640 1 -- 172.21.15.27:0/262714033 >> v1:172.21.15.27:6800/2405211194 conn(0x7f31f00514b0 legacy=0x7f31f00539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:29:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.483+0000 7f321484b640 1 -- 172.21.15.27:0/262714033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32100718e0 msgr2=0x7f321009da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:29:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.483+0000 7f321484b640 1 --2- 172.21.15.27:0/262714033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32100718e0 0x7f321009da70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f320000c460 tx=0x7f320000c920 comp rx=0 tx=0).stop 2024-09-21T03:29:17.484 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.483+0000 7f321484b640 1 -- 172.21.15.27:0/262714033 shutdown_connections 2024-09-21T03:29:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.483+0000 7f321484b640 1 --2- 172.21.15.27:0/262714033 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f3210134eb0 0x7f321009e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.483+0000 7f321484b640 1 --2- 172.21.15.27:0/262714033 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f321012cbd0 0x7f321009dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.483+0000 7f321484b640 1 --2- 172.21.15.27:0/262714033 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f32100718e0 0x7f321009da70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.483+0000 7f321484b640 1 -- 172.21.15.27:0/262714033 >> 172.21.15.27:0/262714033 conn(0x7f32100a1d70 msgr2=0x7f321012d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:29:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.483+0000 7f321484b640 1 -- 172.21.15.27:0/262714033 shutdown_connections 2024-09-21T03:29:17.485 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.483+0000 7f321484b640 1 -- 172.21.15.27:0/262714033 wait complete. 2024-09-21T03:29:17.496 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-09-21T03:29:16.305224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":151,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":116592,"kb_used_data":2288,"kb_used_omap":17,"kb_used_meta":114222,"kb_avail":377370768,"statfs":{"total":386547056640,"available":386427666432,"internally_reserved":0,"allocated":2342912,"data_stored":1580294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17729,"internal_metadata":116964031},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002561"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":60,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:29:00.889829+0000","last_change":"2024-09-21T03:29:00.889829+0000","last_active":"2024-09-21T03:29:00.889829+0000","last_peered":"2024-09-21T03:29:00.889829+0000","last_clean":"2024-09-21T03:29:00.889829+0000","last_became_active":"2024-09-21T03:28:44.675664+0000","last_became_peered":"2024-09-21T03:28:44.675664+0000","last_unstale":"2024-09-21T03:29:00.889829+0000","last_undegraded":"2024-09-21T03:29:00.889829+0000","last_fullsized":"2024-09-21T03:29:00.889829+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:29:00.889715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:29:00.889715+0000","last_clean_scrub_stamp":"2024-09-21T03:29:00.889715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:40:16.422971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001393285,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:56.749794+0000","last_change":"2024-09-21T03:28:56.749794+0000","last_active":"2024-09-21T03:28:56.749794+0000","last_peered":"2024-09-21T03:28:56.749794+0000","last_clean":"2024-09-21T03:28:56.749794+0000","last_became_active":"2024-09-21T03:28:40.835171+0000","last_became_peered":"2024-09-21T03:28:40.835171+0000","last_unstale":"2024-09-21T03:28:56.749794+0000","last_undegraded":"2024-09-21T03:28:56.749794+0000","last_fullsized":"2024-09-21T03:28:56.749794+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:28:56.749716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:28:56.749716+0000","last_clean_scrub_stamp":"2024-09-21T03:28:56.749716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T04:18:38.506628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044937200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":60,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:59.881060+0000","last_change":"2024-09-21T03:28:59.881060+0000","last_active":"2024-09-21T03:28:59.881060+0000","last_peered":"2024-09-21T03:28:59.881060+0000","last_clean":"2024-09-21T03:28:59.881060+0000","last_became_active":"2024-09-21T03:28:44.675680+0000","last_became_peered":"2024-09-21T03:28:44.675680+0000","last_unstale":"2024-09-21T03:28:59.881060+0000","last_undegraded":"2024-09-21T03:28:59.881060+0000","last_fullsized":"2024-09-21T03:28:59.881060+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:28:59.880928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:28:59.880928+0000","last_clean_scrub_stamp":"2024-09-21T03:28:59.880928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T05:23:31.297875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:58.773085+0000","last_change":"2024-09-21T03:28:58.773085+0000","last_active":"2024-09-21T03:28:58.773085+0000","last_peered":"2024-09-21T03:28:58.773085+0000","last_clean":"2024-09-21T03:28:58.773085+0000","last_became_active":"2024-09-21T03:28:40.835243+0000","last_became_peered":"2024-09-21T03:28:40.835243+0000","last_unstale":"2024-09-21T03:28:58.773085+0000","last_undegraded":"2024-09-21T03:28:58.773085+0000","last_fullsized":"2024-09-21T03:28:58.773085+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:28:58.772978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:28:58.772978+0000","last_clean_scrub_stamp":"2024-09-21T03:28:58.772978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T06:07:51.071048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049584300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"34'3","reported_seq":59,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:57.867435+0000","last_change":"2024-09-21T03:28:57.867435+0000","last_active":"2024-09-21T03:28:57.867435+0000","last_peered":"2024-09-21T03:28:57.867435+0000","last_clean":"2024-09-21T03:28:57.867435+0000","last_became_active":"2024-09-21T03:28:43.851555+0000","last_became_peered":"2024-09-21T03:28:43.851555+0000","last_unstale":"2024-09-21T03:28:57.867435+0000","last_undegraded":"2024-09-21T03:28:57.867435+0000","last_fullsized":"2024-09-21T03:28:57.867435+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'3","last_scrub_stamp":"2024-09-21T03:28:57.867313+0000","last_deep_scrub":"34'3","last_deep_scrub_stamp":"2024-09-21T03:28:57.867313+0000","last_clean_scrub_stamp":"2024-09-21T03:28:57.867313+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:59.259325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0010094349999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:56.730683+0000","last_change":"2024-09-21T03:28:56.730683+0000","last_active":"2024-09-21T03:28:56.730683+0000","last_peered":"2024-09-21T03:28:56.730683+0000","last_clean":"2024-09-21T03:28:56.730683+0000","last_became_active":"2024-09-21T03:28:43.850058+0000","last_became_peered":"2024-09-21T03:28:43.850058+0000","last_unstale":"2024-09-21T03:28:56.730683+0000","last_undegraded":"2024-09-21T03:28:56.730683+0000","last_fullsized":"2024-09-21T03:28:56.730683+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:28:56.730592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:28:56.730592+0000","last_clean_scrub_stamp":"2024-09-21T03:28:56.730592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:26:31.483624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056217599999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:56.878041+0000","last_change":"2024-09-21T03:28:56.878041+0000","last_active":"2024-09-21T03:28:56.878041+0000","last_peered":"2024-09-21T03:28:56.878041+0000","last_clean":"2024-09-21T03:28:56.878041+0000","last_became_active":"2024-09-21T03:28:43.851256+0000","last_became_peered":"2024-09-21T03:28:43.851256+0000","last_unstale":"2024-09-21T03:28:56.878041+0000","last_undegraded":"2024-09-21T03:28:56.878041+0000","last_fullsized":"2024-09-21T03:28:56.878041+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T03:28:56.877931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T03:28:56.877931+0000","last_clean_scrub_stamp":"2024-09-21T03:28:56.877931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T08:17:44.577435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045280299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"34'2","reported_seq":58,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:58.891488+0000","last_change":"2024-09-21T03:28:58.891488+0000","last_active":"2024-09-21T03:28:58.891488+0000","last_peered":"2024-09-21T03:28:58.891488+0000","last_clean":"2024-09-21T03:28:58.891488+0000","last_became_active":"2024-09-21T03:28:43.849169+0000","last_became_peered":"2024-09-21T03:28:43.849169+0000","last_unstale":"2024-09-21T03:28:58.891488+0000","last_undegraded":"2024-09-21T03:28:58.891488+0000","last_fullsized":"2024-09-21T03:28:58.891488+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'2","last_scrub_stamp":"2024-09-21T03:28:58.891375+0000","last_deep_scrub":"34'2","last_deep_scrub_stamp":"2024-09-21T03:28:58.891375+0000","last_clean_scrub_stamp":"2024-09-21T03:28:58.891375+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:19.316762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"34'105","reported_seq":210,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T03:28:57.790669+0000","last_change":"2024-09-21T03:28:57.790669+0000","last_active":"2024-09-21T03:28:57.790669+0000","last_peered":"2024-09-21T03:28:57.790669+0000","last_clean":"2024-09-21T03:28:57.790669+0000","last_became_active":"2024-09-21T03:28:40.834765+0000","last_became_peered":"2024-09-21T03:28:40.834765+0000","last_unstale":"2024-09-21T03:28:57.790669+0000","last_undegraded":"2024-09-21T03:28:57.790669+0000","last_fullsized":"2024-09-21T03:28:57.790669+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"34'105","last_scrub_stamp":"2024-09-21T03:28:57.790576+0000","last_deep_scrub":"34'105","last_deep_scrub_stamp":"2024-09-21T03:28:57.790576+0000","last_clean_scrub_stamp":"2024-09-21T03:28:57.790576+0000","objects_scrubbed":2,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:36.819593+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":267,"num_write":149,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":31,"seq":133143986184,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":284,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":290816,"data_stored":99899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4433,"internal_metadata":28176047},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":29,"seq":124554051593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":280,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":286720,"data_stored":99880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32560,"kb_used_data":864,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94339280,"statfs":{"total":96636764160,"available":96603422720,"internally_reserved":0,"allocated":884736,"data_stored":690267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":32435888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":27,"seq":115964117001,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28396,"kb_used_data":860,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94343444,"statfs":{"total":96636764160,"available":96607686656,"internally_reserved":0,"allocated":880640,"data_stored":690248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28176048},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T03:29:17.498 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-21T03:29:17.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.611+0000 7f825a317640 1 -- 172.21.15.27:0/1597685020 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f82540704b0 msgr2=0x7f82540708c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:29:17.612 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.611+0000 7f825a317640 1 --2- 172.21.15.27:0/1597685020 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f82540704b0 0x7f82540708c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8248009860 tx=0x7f824802a460 comp rx=0 tx=0).stop 2024-09-21T03:29:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.611+0000 7f825a317640 1 -- 172.21.15.27:0/1597685020 shutdown_connections 2024-09-21T03:29:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.611+0000 7f825a317640 1 --2- 172.21.15.27:0/1597685020 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f82540704b0 0x7f82540708c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.611+0000 7f825a317640 1 --2- 172.21.15.27:0/1597685020 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8254071c00 0x7f825406ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.611+0000 7f825a317640 1 -- 172.21.15.27:0/1597685020 >> 172.21.15.27:0/1597685020 conn(0x7f825409b180 msgr2=0x7f8254098400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:29:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.611+0000 7f825a317640 1 -- 172.21.15.27:0/1597685020 shutdown_connections 2024-09-21T03:29:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.611+0000 7f825a317640 1 -- 172.21.15.27:0/1597685020 wait complete. 2024-09-21T03:29:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.612+0000 7f825a317640 1 Processor -- start 2024-09-21T03:29:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.612+0000 7f825a317640 1 -- start start 2024-09-21T03:29:17.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f825a317640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f82540704b0 0x7f825412bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:29:17.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f825a317640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8254071c00 0x7f825412c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:29:17.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f825a317640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8254097070 0x7f82541c3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T03:29:17.615 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f825a317640 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 -- 0x7f8254134340 con 0x7f8254097070 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f825a317640 1 -- --> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] -- mon_getmap magic: 0 -- 0x7f825406fd80 con 0x7f82540704b0 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f825a317640 1 -- --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_getmap magic: 0 -- 0x7f82541344c0 con 0x7f8254071c00 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f8253fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f82540704b0 0x7f825412bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f82537fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8254071c00 0x7f825412c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f82588b4640 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8254097070 0x7f82541c3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f8253fff640 1 --2- >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f82540704b0 0x7f825412bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3301/0 says I am v2:172.21.15.27:58862/0 (socket says 172.21.15.27:58862) 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f8253fff640 1 -- 172.21.15.27:0/30152506 learned_addr learned my addr 172.21.15.27:0/30152506 (peer_addr_for_me v2:172.21.15.27:0/0) 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f82537fe640 1 --2- >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8254071c00 0x7f825412c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3302/0 says I am v2:172.21.15.27:44766/0 (socket says 172.21.15.27:44766) 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f82537fe640 1 -- 172.21.15.27:0/30152506 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f82540704b0 msgr2=0x7f825412bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f82537fe640 1 --2- 172.21.15.27:0/30152506 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f82540704b0 0x7f825412bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f82537fe640 1 -- 172.21.15.27:0/30152506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8254097070 msgr2=0x7f82541c3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f82537fe640 1 --2- 172.21.15.27:0/30152506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8254097070 0x7f82541c3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f82537fe640 1 -- 172.21.15.27:0/30152506 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825412c580 con 0x7f8254071c00 2024-09-21T03:29:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f8253fff640 1 --2- 172.21.15.27:0/30152506 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f82540704b0 0x7f825412bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T03:29:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.613+0000 7f82537fe640 1 --2- 172.21.15.27:0/30152506 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8254071c00 0x7f825412c040 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8248031000 tx=0x7f8248033c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T03:29:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.614+0000 7f8250ff9640 1 -- 172.21.15.27:0/30152506 <== mon.2 v2:172.21.15.27:3302/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248005c90 con 0x7f8254071c00 2024-09-21T03:29:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.614+0000 7f8250ff9640 1 -- 172.21.15.27:0/30152506 <== mon.2 v2:172.21.15.27:3302/0 2 ==== config(5 keys) ==== 141+0+0 (secure 0 0 0) 0x7f8248033e10 con 0x7f8254071c00 2024-09-21T03:29:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.614+0000 7f825a317640 1 -- 172.21.15.27:0/30152506 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825412aa20 con 0x7f8254071c00 2024-09-21T03:29:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.614+0000 7f8250ff9640 1 -- 172.21.15.27:0/30152506 <== mon.2 v2:172.21.15.27:3302/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248005280 con 0x7f8254071c00 2024-09-21T03:29:17.617 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.614+0000 7f825a317640 1 -- 172.21.15.27:0/30152506 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f82541c4080 con 0x7f8254071c00 2024-09-21T03:29:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.615+0000 7f825a317640 1 -- 172.21.15.27:0/30152506 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82540994d0 con 0x7f8254071c00 2024-09-21T03:29:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.615+0000 7f8250ff9640 1 -- 172.21.15.27:0/30152506 <== mon.2 v2:172.21.15.27:3302/0 4 ==== mgrmap(e 9) ==== 63498+0+0 (secure 0 0 0) 0x7f8248005420 con 0x7f8254071c00 2024-09-21T03:29:17.618 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.615+0000 7f8250ff9640 1 -- 172.21.15.27:0/30152506 <== mon.2 v2:172.21.15.27:3302/0 5 ==== osd_map(34..34 src has 1..34) ==== 3027+0+0 (secure 0 0 0) 0x7f8248089f30 con 0x7f8254071c00 2024-09-21T03:29:17.620 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.619+0000 7f8250ff9640 1 -- 172.21.15.27:0/30152506 <== mon.2 v2:172.21.15.27:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+158489 (secure 0 0 0) 0x7f82540994d0 con 0x7f8254071c00 2024-09-21T03:29:17.779 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.778+0000 7f825a317640 1 -- 172.21.15.27:0/30152506 --> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f825409cae0 con 0x7f8254071c00 2024-09-21T03:29:17.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.787+0000 7f8250ff9640 1 -- 172.21.15.27:0/30152506 <== mon.2 v2:172.21.15.27:3302/0 7 ==== config(6 keys) ==== 172+0+0 (secure 0 0 0) 0x7f82480317a0 con 0x7f8254071c00 2024-09-21T03:29:17.789 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.788+0000 7f8250ff9640 1 -- 172.21.15.27:0/30152506 <== mon.2 v2:172.21.15.27:3302/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v28)=0 v28) ==== 123+0+0 (secure 0 0 0) 0x7f825409cae0 con 0x7f8254071c00 2024-09-21T03:29:17.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.790+0000 7f825a317640 1 -- 172.21.15.27:0/30152506 >> v1:172.21.15.27:6800/2405211194 conn(0x7f82400514e0 legacy=0x7f82400539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:29:17.791 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.790+0000 7f825a317640 1 -- 172.21.15.27:0/30152506 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8254071c00 msgr2=0x7f825412c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T03:29:17.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.790+0000 7f825a317640 1 --2- 172.21.15.27:0/30152506 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8254071c00 0x7f825412c040 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8248031000 tx=0x7f8248033c20 comp rx=0 tx=0).stop 2024-09-21T03:29:17.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.790+0000 7f825a317640 1 -- 172.21.15.27:0/30152506 shutdown_connections 2024-09-21T03:29:17.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.790+0000 7f825a317640 1 --2- 172.21.15.27:0/30152506 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8254097070 0x7f82541c3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.790+0000 7f825a317640 1 --2- 172.21.15.27:0/30152506 >> [v2:172.21.15.27:3302/0,v1:172.21.15.27:6791/0] conn(0x7f8254071c00 0x7f825412c040 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.790+0000 7f825a317640 1 --2- 172.21.15.27:0/30152506 >> [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] conn(0x7f82540704b0 0x7f825412bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T03:29:17.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.790+0000 7f825a317640 1 -- 172.21.15.27:0/30152506 >> 172.21.15.27:0/30152506 conn(0x7f825409b180 msgr2=0x7f8254132b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T03:29:17.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.790+0000 7f825a317640 1 -- 172.21.15.27:0/30152506 shutdown_connections 2024-09-21T03:29:17.792 INFO:teuthology.orchestra.run.smithi027.stderr:2024-09-21T03:29:17.790+0000 7f825a317640 1 -- 172.21.15.27:0/30152506 wait complete. 2024-09-21T03:29:17.803 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-21T03:29:17.803 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-21T03:29:17.804 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-21T03:29:17.804 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:29:17.833 INFO:tasks.ceph.osd.0:Stopped 2024-09-21T03:29:17.834 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-21T03:29:17.834 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:29:17.904 INFO:tasks.ceph.osd.1:Stopped 2024-09-21T03:29:17.904 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-21T03:29:17.904 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:29:17.977 INFO:tasks.ceph.osd.2:Stopped 2024-09-21T03:29:17.978 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-21T03:29:17.978 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:29:18.051 INFO:tasks.ceph.osd.3:Stopped 2024-09-21T03:29:18.052 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-21T03:29:18.052 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-21T03:29:18.052 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:29:18.114 INFO:tasks.ceph.mgr.x:Stopped 2024-09-21T03:29:18.114 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-21T03:29:18.114 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:29:18.179 INFO:tasks.ceph.mgr.y:Stopped 2024-09-21T03:29:18.179 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-21T03:29:18.179 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-21T03:29:18.179 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:29:18.231 INFO:tasks.ceph.mon.a:Stopped 2024-09-21T03:29:18.231 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-21T03:29:18.231 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:29:18.283 INFO:tasks.ceph.mon.b:Stopped 2024-09-21T03:29:18.283 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-21T03:29:18.283 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T03:29:18.335 INFO:tasks.ceph.mon.c:Stopped 2024-09-21T03:29:18.335 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-21T03:29:18.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'overall HEALTH_' | egrep -v '\(FS_' | egrep -v '\(MDS_' | egrep -v '\(OSD_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(CACHE_POOL_' | egrep -v '\(POOL_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'Monitor daemon marked osd' | egrep -v 'Behind on trimming' | egrep -v 'Manager daemon' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'scrub mismatch' | egrep -v ScrubResult | egrep -v 'wrongly marked' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_MSGR2_NOT_ENABLED\)' | egrep -v 'slow request' | head -n 1 2024-09-21T03:29:18.426 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:29:18.426 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-21T03:29:18.610 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:29:18.611 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-21T03:29:18.668 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:29:18.669 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-21T03:29:18.736 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:29:18.736 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-21T03:29:18.795 INFO:tasks.ceph:Archiving mon data... 2024-09-21T03:29:18.795 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913465/data/mon.a.tgz 2024-09-21T03:29:18.796 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-09-21T03:29:18.811 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.LTTccWg0j6 2024-09-21T03:29:18.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.LTTccWg0j6 2024-09-21T03:29:18.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.LTTccWg0j6 2024-09-21T03:29:19.038 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.LTTccWg0j6 is 237KB 2024-09-21T03:29:19.104 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.LTTccWg0j6 2024-09-21T03:29:19.120 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913465/data/mon.b.tgz 2024-09-21T03:29:19.120 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-09-21T03:29:19.174 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.zMEX2n95Ta 2024-09-21T03:29:19.175 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.zMEX2n95Ta 2024-09-21T03:29:19.292 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.zMEX2n95Ta 2024-09-21T03:29:19.377 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.zMEX2n95Ta is 281KB 2024-09-21T03:29:19.471 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.zMEX2n95Ta 2024-09-21T03:29:19.486 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913465/data/mon.c.tgz 2024-09-21T03:29:19.486 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2024-09-21T03:29:19.541 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.eO8Vp7S2uw 2024-09-21T03:29:19.542 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.eO8Vp7S2uw 2024-09-21T03:29:19.660 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.eO8Vp7S2uw 2024-09-21T03:29:19.744 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.eO8Vp7S2uw is 281KB 2024-09-21T03:29:19.839 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.eO8Vp7S2uw 2024-09-21T03:29:19.854 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-21T03:29:19.855 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T03:29:19.976 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T03:29:19.976 INFO:tasks.ceph:Archiving crash dumps... 2024-09-21T03:29:19.978 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913465/remote/smithi027/crash 2024-09-21T03:29:19.978 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T03:29:20.041 INFO:tasks.ceph:Compressing logs... 2024-09-21T03:29:20.041 DEBUG:teuthology.orchestra.run.smithi027:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T03:29:20.106 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-21T03:29:20.106 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39740.log 2024-09-21T03:29:20.145 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33761.log 2024-09-21T03:29:20.145 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.39740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46813.log 2024-09-21T03:29:20.145 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33761.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39740.log.gz 2024-09-21T03:29:20.145 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46943.log 2024-09-21T03:29:20.145 INFO:teuthology.orchestra.run.smithi027.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.33761.log.gz 2024-09-21T03:29:20.145 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47427.log 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46813.log: /var/log/ceph/ceph-client.admin.46943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46564.log 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.46813.log.gz 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.46943.log.gz 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42586.log 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46564.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.47427.log.gz 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39556.log 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.46564.log.gz 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46420.log 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42586.log.gz 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46814.log 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39556.log.gz 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32955.log 2024-09-21T03:29:20.146 INFO:teuthology.orchestra.run.smithi027.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.46420.log.gz 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40037.log 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.46814.log.gz 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47111.log 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40037.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.32955.log.gz 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41836.log 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40037.log.gz 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46629.log 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.47111.log.gz 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46298.log 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41836.log.gz 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42910.log 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.46629.log.gz 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33145.log 2024-09-21T03:29:20.147 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.46298.log.gz 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46460.log 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42910.log.gz 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33145.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.33145.log.gz 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42544.log 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46460.log.gz 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47971.log 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41829.log 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.42544.log.gz 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33261.log 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.47971.log.gz 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41829.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41829.log.gz 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46587.log 2024-09-21T03:29:20.148 INFO:teuthology.orchestra.run.smithi027.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.33261.log.gz 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40396.log 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41656.log: /var/log/ceph/ceph-client.admin.46587.log: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47151.log 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.46587.log.gz 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46812.log 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40396.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40396.log.gz 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.47151.log.gz 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42197.log 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46812.log: /var/log/ceph/ceph-client.admin.39909.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.46812.log.gz 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46500.log 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2024-09-21T03:29:20.149 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47721.log 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42197.log: /var/log/ceph/ceph-client.admin.46500.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42197.log.gz 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42670.log 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.46500.log.gz 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47805.log 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47721.log: /var/log/ceph/ceph-client.admin.42670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42277.log 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.47721.log.gz 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47805.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42670.log.gz 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40312.log 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.47805.log.gz 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33155.log 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42277.log.gz 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39514.log 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40312.log: /var/log/ceph/ceph-client.admin.33155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32989.log 2024-09-21T03:29:20.150 INFO:teuthology.orchestra.run.smithi027.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40312.log.gz 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.33155.log.gz 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33052.log 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39514.log.gz 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33189.log 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32989.log: /var/log/ceph/ceph-client.admin.33052.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.32989.log.gz 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47343.log 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.33052.log.gz 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41517.log 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.33189.log.gz 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46815.log 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47343.log: /var/log/ceph/ceph-client.admin.41517.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.47343.log.gz 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32493.log 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41517.log.gz 2024-09-21T03:29:20.151 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33566.log 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46815.log: /var/log/ceph/ceph-client.admin.32493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45060.log 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32493.log.gz 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.46815.log.gz 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47385.log 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.33566.log.gz 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.45060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41765.log 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.45060.log.gz 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39741.log 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47385.log: /var/log/ceph/ceph-client.admin.41765.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.47385.log.gz 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39640.log 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41765.log.gz 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33251.log 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39741.log: /var/log/ceph/ceph-client.admin.39640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33801.log 2024-09-21T03:29:20.152 INFO:teuthology.orchestra.run.smithi027.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39741.log.gz 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39640.log.gz 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33224.log 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.33251.log.gz 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39907.log 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.33801.log.gz 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46713.log 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33224.log: /var/log/ceph/ceph-client.admin.39907.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.33224.log.gz 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39907.log.gz 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33680.log 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46713.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33032.log 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.46713.log.gz 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47553.log 2024-09-21T03:29:20.153 INFO:teuthology.orchestra.run.smithi027.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.33680.log.gz 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39742.log 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr: 85.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.33032.log.gz 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41434.log 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47553.log: /var/log/ceph/ceph-client.admin.39742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42417.log 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.47553.log.gz 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39742.log.gz 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32791.log 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr: 95.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41434.log.gz 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34087.log 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42417.log.gz 2024-09-21T03:29:20.154 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32735.log 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32791.log: 76.4% -- replaced with /var/log/ceph/ceph-client.admin.32791.log.gz 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.34087.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34087.log.gz 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32470.log 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32735.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32735.log.gz 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39739.log 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32470.log.gz 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40038.log 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42743.log 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39739.log: /var/log/ceph/ceph-client.admin.40038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40354.log 85.7% 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39739.log.gz 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40038.log.gz 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42217.log 2024-09-21T03:29:20.155 INFO:teuthology.orchestra.run.smithi027.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42743.log.gz 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46983.log 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr:gzip 86.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.40354.log.gz --verbose 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr: -- /var/log/ceph/ceph-client.admin.40186.log 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42217.log: /var/log/ceph/ceph-client.admin.46983.log: gzip -5 86.9% --verbose -- replaced with /var/log/ceph/ceph-client.admin.42217.log.gz -- 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr: /var/log/ceph/ceph-client.admin.41658.log 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46983.log.gz 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32763.log 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39910.log 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40186.log.gz 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41658.log: /var/log/ceph/ceph-client.admin.32763.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.41658.log.gz 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.42456.log 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32763.log.gz 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47511.log 2024-09-21T03:29:20.156 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42456.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32699.log 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42456.log.gz 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32893.log 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47889.log 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.47511.log.gz 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32699.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32699.log.gz 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41839.log 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr: 75.6% -- replaced with /var/log/ceph/ceph-client.admin.32893.log.gz 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47889.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.47889.log.gz 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40077.log 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41839.log.gz 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47931.log 2024-09-21T03:29:20.157 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42043.log 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40077.log.gz 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47637.log 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42043.log: 86.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46541.log 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47931.log.gz 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42043.log.gz 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47847.log 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.47637.log.gz 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42376.log 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr: 87.2%/var/log/ceph/ceph-client.admin.47847.log: -- replaced with /var/log/ceph/ceph-client.admin.46541.log.gz 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47259.log 86.5% 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47847.log.gz 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33721.log 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42376.log.gz 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-21T03:29:20.158 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47259.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.47259.log.gz 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40078.log 86.8% -- replaced with /var/log/ceph/ceph-client.admin.33721.log.gz 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39463.log 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40078.log.gz 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42118.log 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39463.log: /var/log/ceph/ceph-client.admin.46380.log 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39463.log.gz 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42118.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42118.log.gz 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33102.log 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.46380.log.gz 2024-09-21T03:29:20.159 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.log: 93.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-21T03:29:20.160 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32641.log 2024-09-21T03:29:20.160 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38456.log 2024-09-21T03:29:20.160 INFO:teuthology.orchestra.run.smithi027.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.33102.log.gz 2024-09-21T03:29:20.160 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41968.log 2024-09-21T03:29:20.160 INFO:teuthology.orchestra.run.smithi027.stderr: 77.2% -- replaced with /var/log/ceph/ceph-client.admin.32641.log.gz 2024-09-21T03:29:20.160 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.38456.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38456.log.gz 2024-09-21T03:29:20.160 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33164.log 2024-09-21T03:29:20.160 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41968.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-21T03:29:20.161 INFO:teuthology.orchestra.run.smithi027.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41968.log.gz 2024-09-21T03:29:20.185 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.33164.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47150.log 2024-09-21T03:29:20.185 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33164.log.gz 2024-09-21T03:29:20.185 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mgr.x.log: 91.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40225.log 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47469.log 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.47150.log.gz 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39438.log 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47469.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40225.log.gz 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.47469.log.gz 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33919.log 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42628.log 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39438.log: /var/log/ceph/ceph-client.admin.33919.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39438.log.gz 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.33919.log.gz 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41661.log 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46982.log 2024-09-21T03:29:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41661.log: /var/log/ceph/ceph-client.admin.42628.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42628.log.gz 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41661.log.gz 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42396.log 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39598.log 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46982.log: /var/log/ceph/ceph-client.admin.42396.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.46982.log.gz 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42396.log.gz 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41559.log 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39598.log: /var/log/ceph/ceph-mon.b.log 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39598.log.gz 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41559.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41559.log.gz 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40477.log 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42237.log 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40477.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40477.log.gz 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47595.log 2024-09-21T03:29:20.187 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42237.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42237.log.gz 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40226.log 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47595.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.47595.log.gz 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40436.log 93.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47301.log 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.40436.log: /var/log/ceph/ceph-client.admin.40226.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40226.log.gz 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40436.log.gz 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46671.log 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39382.log 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.47301.log.gz 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42992.log 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.39382.log: /var/log/ceph/ceph-client.admin.46671.log: 87.0% 87.1% -- replaced with /var/log/ceph/ceph-client.admin.46671.log.gz 2024-09-21T03:29:20.188 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39382.log.gz 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39908.log 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32669.log 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42992.log: /var/log/ceph/ceph-client.admin.39908.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42992.log.gz 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39908.log.gz 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47763.log 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.32669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32669.log.gz 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33127.log 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41948.log 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47763.log: /var/log/ceph/ceph-client.admin.33127.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.47763.log.gz 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.33127.log.gz 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47091.log 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41948.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41948.log.gz 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33521.log 2024-09-21T03:29:20.189 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26586.log 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.47091.log: /var/log/ceph/ceph-client.admin.33521.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.47091.log.gz 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.33521.log.gz 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46340.log 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph.tmp-client.admin.26586.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26586.log.gz 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41660.log 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46340.log: /var/log/ceph/ceph-osd.3.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46340.log.gz 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42822.log 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.41660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46935.log 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.42822.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41660.log.gz 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47679.log 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42822.log.gz 2024-09-21T03:29:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:/var/log/ceph/ceph-client.admin.46935.log: /var/log/ceph/ceph-client.admin.47679.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.46935.log.gz 2024-09-21T03:29:20.191 INFO:teuthology.orchestra.run.smithi027.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.47679.log.gz 2024-09-21T03:29:20.193 INFO:teuthology.orchestra.run.smithi027.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-21T03:29:20.204 INFO:teuthology.orchestra.run.smithi027.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-21T03:29:20.206 INFO:teuthology.orchestra.run.smithi027.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-21T03:29:20.231 INFO:teuthology.orchestra.run.smithi027.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-21T03:29:20.245 INFO:teuthology.orchestra.run.smithi027.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-21T03:29:20.262 INFO:teuthology.orchestra.run.smithi027.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-21T03:29:20.264 INFO:teuthology.orchestra.run.smithi027.stderr: 2024-09-21T03:29:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:real 0m0.168s 2024-09-21T03:29:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:user 0m0.653s 2024-09-21T03:29:20.264 INFO:teuthology.orchestra.run.smithi027.stderr:sys 0m0.113s 2024-09-21T03:29:20.264 INFO:tasks.ceph:Archiving logs... 2024-09-21T03:29:20.264 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913465/remote/smithi027/log 2024-09-21T03:29:20.265 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T03:29:20.519 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T03:29:20.535 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T03:29:20.536 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T03:29:20.583 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T03:29:20.593 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T03:29:20.594 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T03:29:20.617 INFO:teuthology.orchestra.run.smithi027.stderr:bash: line 1: ntpq: command not found 2024-09-21T03:29:20.621 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T03:29:20.621 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2024-09-21T03:29:20.621 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv01.front.sepia.ceph.com 3 6 177 63 -231us[ -233us] +/- 43ms 2024-09-21T03:29:20.621 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv02.front.sepia.ceph.com 3 6 177 63 +375us[ +375us] +/- 69ms 2024-09-21T03:29:20.621 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 177 64 -8177ns[ -10us] +/- 74ms 2024-09-21T03:29:20.621 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T03:29:20.622 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T03:29:20.634 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T03:29:20.676 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T03:29:20.687 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T03:29:20.753 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T03:29:20.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T03:29:20.781 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T03:29:20.792 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T03:29:20.802 INFO:teuthology.task.internal:Duration was 492.045094 seconds 2024-09-21T03:29:20.802 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T03:29:20.813 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T03:29:20.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T03:29:20.862 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T03:29:21.256 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T03:29:21.256 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2024-09-21T03:29:21.256 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T03:29:21.318 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T03:29:21.319 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T03:29:21.387 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T03:29:21.387 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T03:29:21.533 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T03:29:21.543 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T03:29:21.544 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T03:29:21.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T03:29:21.580 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T03:29:21.633 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2024-09-21T03:29:21.660 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T03:29:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T03:29:21.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T03:29:21.728 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T03:29:21.728 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913465/remote/smithi027 2024-09-21T03:29:21.729 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T03:29:21.790 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T03:29:21.790 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T03:29:21.840 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T03:29:21.852 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T03:29:21.852 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T03:29:21.863 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T03:29:21.863 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T03:29:21.896 INFO:teuthology.orchestra.run.smithi027.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 03:29 /home/ubuntu/cephtest 2024-09-21T03:29:21.897 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T03:29:21.910 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T03:29:21.935 INFO:teuthology.run:Summary data: description: fs/upgrade/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/squid 1-upgrade}} duration: 492.0450940132141 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-09-21T03:29:21.935 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T03:29:22.022 INFO:teuthology.run:pass